The sparse-model bill arrives at serving time, and it is paid in collectives

The final installment of a five-part CUDA and ROCm tutorial series on dev.to spends its length on four mechanisms rather than one architecture: Mixture of Experts routing with all-to-all expert parallelism, a ZeRO-Offload path that spills optimizer state to system RAM, CUDA/HIP graphs to eliminate kernel launch overhead, and an inference server with KV caching and PagedAttention ^[1]. The author's framing is the useful part: training is only half the battle, serving is the other half, and the prerequisites are Parts 1 through 4 plus a multi-GPU box, four cards or more ^[2].

Start with the arithmetic that forces the design. A dense 1.5 trillion parameter model would need 3TB of VRAM for weights alone ^[4], which works out to two bytes per parameter, so that figure already assumes half precision and gets no rescue from further quantisation of the same kind ^[9]. MoE answers this by holding hundreds of expert feed-forward networks and activating two of them per token ^[5], with the output a softmax-weighted sum over the top-k experts, k usually 2 ^[6]. The author notes that in 2024 and 2025 the major models, naming Grok, Mixtral and Gemini, use this to reach trillions of parameters without a matching compute bill ^[3].

What that buys in FLOPs it spends on the network. Dense models used All-Reduce; MoE needs All-to-All, because different GPUs own different experts and every token has to travel to whichever rank holds the expert it was routed to ^[10]. The listing is three steps: an NCCL/RCCL grouped send and receive built from per-rank send and receive counts, a local expert kernel over whatever arrived, then a second all-to-all to return the processed tokens to their origin GPUs ^[11]. That is two collectives per MoE layer per forward pass, and both are shaped by routing decisions that are not known until the router runs ^[12]. The author asserts this scales to thousands of experts across a cluster with minimal communication overhead ^[13]; that claim is unquantified in the text and is exactly the number an operator should measure rather than accept.

The router kernel itself is honest about being illustrative. It tracks the top two scores in a manual reduction and applies softmax to only those two, which is the right sparse move ^[7]. But it runs one thread per token and loops over every expert and every hidden dimension to get the logits ^[8], so routing cost per token scales with the full expert count regardless of k [c8a].

The offload section targets the other wall. At 70B parameters, ZeRO-3 may not fit in 40GB of VRAM, so momentum, variance and sometimes gradients go to host DDR ^[14], allocated as pinned memory ^[16], with gradients copied out asynchronously during the backward pass, the AdamW update computed on a CPU thread, and FP32 weights copied back just in time for the next forward ^[15]. If those two moment buffers are FP32, as the FP32 master weights imply, they are roughly 560GB on their own, about fourteen times a 40GB card ^[17]. The throughput ceiling of that scheme is PCIe, not the GPU.

Watch the two items the supplied text promises but breaks off before delivering: the graph capture and the paged KV cache both appear only in the agenda, with the article cutting off mid-listing inside the offload code ^[18]. Those are the pieces that decide decode latency, and a tutorial that stops before them has described the problem, not the fix.