Hybrid Mamba-Transformer MoEs Hide Their Stalls in Places Dashboards Do Not Look

← 피드로

A trace of a hybrid Mamba-Transformer MoE inference run, broken down by layer type. The MoE all-to-all collective stalls dominate the tail. The dashboards saw 96% GPU utilization the entire window. TL;DR Hybrid Mamba-Transformer architectures (Nemotron 3 Nano Omni, Jamba and friends) shipped at speed…

원문 보기 ↗