Build1 publisher2 min readPublished
A 30B model that activates 3B still has to keep all 30B in VRAM
NVIDIA's dense-versus-MoE explainer uses Nemotron 3.5 Lightning to walk through per-layer routing. Its own text puts the throughput number and the memory number on two different parameter counts.
The Engineer · Build desk
What happened
- NVIDIA's developer blog explains how Nemotron 3.5 Lightning holds 30B parameters but activates only 3B per token, using a Mixture-of-Experts architecture that selects a subset of parameters for each token.
- In the dense case every parameter runs on every token, and the post's example is a 27B model whose full 27B fires through a single shared feed-forward block in each decoder layer.
- An MoE layer replaces that one shared block with multiple experts, for example 8, 64 or 128, and a learned router sends each token to the top k scorers while the rest sit out that layer.
- The post says the more consequential difference is that MoE decouples memory from compute, with total parameter count determining the VRAM the model has to be hosted in.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Sizing a host still starts from the total: a 30B MoE that activates 3B needs memory for all 30B, so the sparsity saving shows up in per-token compute and not in the GPU you have to buy.
- decision Anyone weighing an MoE against a dense model has to rerun the comparison on their own stack, because NVIDIA ties the result to hardware, precision, inference framework and model design, and Lightning also runs speculative decoding.
- capability The long-context memory behaviour NVIDIA attributes to Mamba-2's constant-size recurrent state travels with the hybrid design, so copying the MoE part alone does not buy it.
Two counts govern the deployment, and NVIDIA's post keeps them apart. Total parameters set what has to sit in VRAM. Active parameters set what runs on each token. For a 30B model that activates 3B, about 10 percent of the weights do work on any given token, and the other 27B stay resident and idle for it [1][19].
What the 3B contains matters for sizing. NVIDIA says a "3B active parameters" figure on a model card includes attention and embedding weights for every token along with the selected feed-forward weights [11], and that tokens pass through the full attention mechanism as normal no matter how the router decides [10]. The routed expert share is therefore below 3B [20], though the post does not break out the split.
The routing happens per layer, not per request. A learned gate network sits in front of the experts and assigns each token to the top k scorers; the blocks that lose run nothing for that token at that layer [6]. Where a dense model has one shared feed-forward block per decoder layer, an MoE layer may have 8, 64 or 128 [4][5]. The decision is made again at every layer, based on what the token represents at that point in the network [8]. The experts' specializations lie primarily in syntax and token-type patterns such as punctuation and numbers, and vary by architecture or training method, according to the post [9]. Most modern MoEs, Mistral Small 4 among them, also keep one shared expert that every token is routed to regardless [7].
For the throughput claim to transfer to another fleet, several things have to hold, and NVIDIA's own caveats are the checklist. At high concurrency, routing and memory movement can narrow MoE's advantage [15]. Results depend on hardware, precision, inference framework and model design, and Lightning runs speculative decoding [16]. A number measured on that setup is a claim about that configuration. Dense models activate the whole network but offer simpler serving and more predictable latency [14]. Adopting routing costs that predictability.
Sparsity accounts for only part of Lightning's profile. The NVIDIA model card specifies a Mamba-2 + MoE + Attention hybrid, in which Mamba-2 layers stand in for attention in most layers and carry a constant-size recurrent state rather than a growing KV cache; NVIDIA writes that this changes the memory profile at long context in a way sparsity alone does not account for [12]. Lightning also compresses its routing decisions into a smaller space first, instead of making them across the full model's breadth, to make the decision cheaper [13].
What to watch
- Whether NVIDIA publishes concurrency and batch-size figures for Lightning, so the throughput claim can be checked at the load a production fleet actually runs.
- Whether the long-context memory behaviour attributed to Mamba-2 gets measured separately from MoE sparsity in any follow-up.
- Whether model cards begin listing the routed feed-forward share alongside total and active parameter counts.