other

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

2017 Google Brain paper that introduced sparse top-k gating for mixture-of-experts layers, along with the auxiliary loss used to keep expert usage balanced.

Known aliases

Shazeer et al. 2017
Sparsely-Gated Mixture-of-Experts Layer

Relationships

No evidence-backed relationships are recorded.

Current clusters

build1 publisher

Top-k gating decouples DeepSeek-V3's 671B parameter count from its 37B of per-token compute

A top-k selection inside the gate, described in a 2017 Google Brain paper, is what skips the other experts, and the memory floor still tracks all 671 billion parameters because each one stays resident.

Publishers:dev.to

Reality

Evidence60
Adoption38
Hype gap0
Incentives20
Confidence55