NVIDIA's unoptimized DeepSeek-V3 baseline spends 84% of kernel time moving tokens between GPUs

The shapes are ragged because the router learns. NVIDIA's post says the expert distribution can become heavily skewed as the router develops preferences for certain experts, and that no two batches produce the same expert loads ^[4]. Each expert receives a different number of tokens, so there is no clean rectangular GEMM to batch and dispatch ^[5]. Most libraries are highly optimized for tensor operations that expect uniform, rectangular data structures ^[6].

Under expert parallelism, that irregularity turns into network traffic. Tokens have to be dispatched to the GPU holding their expert, and the outputs have to be combined and restored to the original token order ^[7]. "A poorly optimized all-to-all forces GPUs to stall and wait for data before doing any useful work," the post says ^[8].

If communication is 84% of accumulated kernel time, compute is the other 16%, so deleting every byte of traffic and leaving compute alone gives at most 1/0.16, about 6.25x ^[17]. The reported gain is 10.4x, and 1,068 divided by 103 confirms it ^[16]. The extra factor has to come from the compute side. Two of the three building blocks NVIDIA lists are on that side: group-aware MXFP8 quantization and an MXFP8 grouped GEMM on the expert matmuls, against one entry for optimized dispatch and combine ^[9][18]. The 84% is a share of accumulated kernel time, not wall clock ^[2], so 6.25x is an approximate ceiling.

For the 10.4x to transfer, the DeepSeek-V3 configuration, GB200 hardware, dropless routing under expert parallelism, and the JAX plus Transformer Engine path all have to match ^[3]. So does the starting point.

Committing to dropless carries a cost the throughput figure does not show. Every kernel that touches expert computation has to handle variable token counts, and it has to work when those shapes are inaccessible on the CPU, so that CUDA graphs stay usable and the stack avoids recompilation ^[11]. The alternative is capacity-based routing, which gives each expert a fixed token budget and trims or pads the overflow: regular shapes, paid for either in dropped training data or in wasted compute and memory ^[12]. MegaBlocks got around that by reformulating expert computation as block-sparse matrix multiplication, which lets each expert take a different number of tokens without dropping or padding. The bill is new block-sparse kernels, an optimized grouped GEMM, and dispatch and combine primitives written for variable token counts ^[13].

The same pattern runs through the model family. DeepSeek, Qwen and Mixtral all match or exceed dense counterparts at a fraction of the training compute ^[15], and all of them inherit the same four bottlenecks that dense models do not have: token routing, expert dispatch and gather, all-to-all communication, and ragged expert GEMMs ^[14]. What matters for a given cluster is the share of its own kernel time sitting in dispatch and combine. On the run NVIDIA published, that share was 84% before the kernel work ^[2].