Top-k gating decouples DeepSeek-V3's 671B parameter count from its 37B of per-token compute

Call a sparse mixture-of-experts layer and the first thing that runs is one matrix multiply: `logits = x @ gate_weights`, a raw score per expert. Then `topk` takes the k highest, softmax renormalises over just those k, and the layer runs only the selected experts, weighting each output by its renormalised score ^[9]. In Shazeer et al.'s 2017 paper from Google Brain, k was 4 and the pool held 4,096 experts ^[8]. That is one expert in 1,024, about 0.1 percent of the pool per token ^[14].

DeepSeek-V3 sits at a milder ratio. It holds 671 billion parameters and uses roughly 37 billion of them each time, which is 5.5 percent activated and an 18-fold gap between what is stored and what runs ^[3][13].

Whether that gap turns into money depends on what you are paying for. In a dense model every parameter activates on every token, and the compute of a forward pass scales about linearly with parameter count ^[1]. Sparse routing breaks that link on the compute side, so the 18x is a FLOPs ratio. Converting it into a lower cost per GPU-hour needs enough concurrent traffic that the resident weights are amortised across a lot of tokens. The model still carries hundreds of billions of parameters in vRAM while activating a small fraction per run ^[4].

The routing idea is older than the saving. Jacobs, Jordan, Nowlan and Hinton's 1991 paper, Adaptive Mixtures of Local Experts, trained several small expert networks alongside a separate gating network that learned to weight and combine their outputs, all jointly ^[5]. The test was vowel discrimination, competing experts plus a gate against one monolithic network, and the learning procedure decomposed the hard problem into sub-problems each expert could handle ^[6]. The gate emitted a weight for every expert and every expert computed on every input, so nothing was skipped ^[7].

What breaks is the gate itself, during training. A small random edge at initialisation makes one expert respond marginally better to some input, and the gate routes slightly more traffic there. That expert collects more gradient updates than its neighbours, improves, and the gate favours it further ^[10]. The 2017 fix adds an auxiliary loss on top of the task loss, penalising uneven expert usage across a batch ^[11]. It is computed as the expert count times the sum over experts of routed fraction times average gate probability ^[12]. Both factors are batch statistics, so the penalty is only as informative as the batch is large ^[12].

The write-up does not give serving costs or latency for any of these models. Per-token compute follows the 37 billion, and the hardware you have to buy follows the 671 billion ^[3][4].