A four-item MoE agenda, two items delivered, zero numbers: reading Part 5 honestly

The final part of the dev.to series "Advanced GPU Optimization: How to tech an LLM with CUDA and ROCm?" sets out four deliverables: Mixture of Experts routing with expert parallelism over all-to-all, a ZeRO-Offload path that spills optimizer states to system RAM, CUDA/HIP graphs to remove kernel launch overhead, and an inference server with KV caching and PagedAttention ^[1]. The supplied text ends mid-listing inside the ZeRO-Offload training step, at the word "backward", so the graph and PagedAttention sections exist only as agenda items ^[18]. That matters because those two are the serving half the piece itself names as the point, listing among its prerequisites "a realization that training is only half the battle - serving is the other half" ^[2].

Take the framing first. The author says every major model in 2024/2025, naming Grok, Mixtral and Gemini, uses MoE to reach trillions of parameters without exploding compute ^[3], and anchors it on a dense 1.5 trillion parameter model needing 3TB of VRAM for weights ^[4]. That arithmetic implies 2 bytes per parameter, so it is a half-precision figure, not a worst case ^[9]. MoE is then presented as hundreds of expert FFNs with only 2 active per token ^[5], following the standard top-k gate with k usually 2 ^[6].

Now the code. The published router kernel assigns one thread per token and loops over every expert, and inside that over every embedding dimension, to compute the logits, tracking the top two by manual reduction and applying softmax only to those two scores ^[7][8]. Routing work per token is therefore proportional to num_experts times D and does not shrink with k [c8a]. That is not a defect in a teaching kernel, but it is the opposite of the sparsity story the section is selling: the saving lives in the FFNs, not in the gate.

The communication section is where the gap is widest. Dense training uses All-Reduce; MoE needs All-to-All because experts live on different GPUs and tokens must travel to their owner ^[10]. The implementation is a dispatch sized by per-rank send and receive counts, a local expert kernel, and a second all-to-all to return results ^[11], which is two collectives per MoE layer per forward pass with message sizes set by routing decisions ^[12]. On that basis the author asserts the scheme "ensures we scale to thousands of experts across a cluster with minimal communication overhead" ^[13]. No timing accompanies it in the supplied text.

The offload section has the same shape. At 70B parameters, ZeRO-3 may not fit in 40GB and optimizer states move to CPU DDR ^[14]; FP32 momentum plus variance for 70B is about 560GB, roughly fourteen 40GB GPUs' worth ^[17]. The mechanism is asynchronous gradient copies, an AdamW update on a CPU thread, and FP32 weights copied back in time for the next forward ^[15], with buffers in pinned memory via hipHostMalloc ^[16]. Whether that overlap holds is entirely a PCIe bandwidth question, and no bandwidth figure appears.

Read the incentives plainly: a serialized tutorial is rewarded for agenda breadth and for shipping a part labelled final, not for the benchmark that would falsify a phrase like "minimal communication overhead" ^[1][13]. Operating consequence: treat Part 5 as a structural map, not an implementation you can cost. If you are planning expert parallelism from it, budget the two collectives per layer yourself ^[12] and profile the router's expert-by-dimension scan before you trust the sparsity accounting [c8a].