Build1 publisher2 min readPublished
Muon throws away a gradient's singular values and keeps its directions
The optimizer behind an October 2024 NanoGPT speed record is now part of the mainstream PyTorch stack. Its scope covers hidden matrix parameters, so the rest of the model still needs a second optimizer.
The Engineer · Build desk
What happened
- Muon's core step takes the momentum update, keeps its singular directions, discards the singular-value magnitudes, and approximates that operation with a few Newton-Schulz iterations.
- After faster training in small-model competitions and runs at multi-billion-parameter scale, Muon is now part of the mainstream PyTorch optimization stack, according to the dev.to write-up.
- Keller Jordan and collaborators published the idea during the NanoGPT speedrunning competition in October 2024, and a Muon-based run on October 15 improved the previous training-speed record by about 35%.
- Vlado Boza found that treating Q, K and V separately worked better, and Yuchen Jin pushed the experiments to larger models while supplying much of the H100 compute.
- AdamW keeps its second-moment estimate elementwise, so a 4096 x 4096 weight matrix is roughly 16 million separate scalar coordinates to it.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- decision Muon's scope stops at hidden matrix parameters. Adopting it means writing a parameter-group split into the training config and defending that boundary at review.
- cost Every matrix parameter now pays for a few Newton-Schulz iterations each step, so the training-time win has to be larger than the matmul bill the orthogonalization adds.
- constraint The headline margin was measured on a small-model speedrun. That bounds what an engineer can promise internally before running the same comparison at their own model size and step budget.
- precedent A leaderboard built around one tiny training program has now fed a mainstream optimizer stack. The next default may arrive by the same route.
The write-up's worked example has an update with singular values 20, 3 and 0.2 [10]. The largest is 100 times the smallest [11]. Muon keeps the same singular vectors and replaces the magnitudes with an approximation of all ones, scaled for the matrix shape [10][9]. The direction that was moving at one hundredth of the dominant one now moves alongside it. Whether that helps depends on what lives in the small direction: signal that AdamW was starving, or noise you have just promoted by two orders of magnitude.
AdamW never forms that ratio. Its second moment is elementwise [4], so a single 4096 x 4096 layer is 16,777,216 independent coordinates [7], and the two moving averages in the update rule come to about 33.5 million scalar statistics for that one layer [6][8]. None of those statistics describe the layer as a transformation [4].
Muon is defined for the hidden matrix parameters of a network [1]. Anything that is not a hidden matrix still needs an optimizer, so a training script that adopts Muon picks up a parameter-group split, and someone has to own where the line falls. The post covers the operation and the provenance. The PyTorch API, the defaults and a suggested split are not there [22].
The unit you orthogonalize is a modelling decision as much as an optimizer setting. Vlado Boza found that treating Q, K and V separately worked better [15]. If your attention block stores one fused QKV projection, orthogonalizing that single matrix is a different operation, and in my view that is worth checking in an existing codebase before anyone compares loss curves.
The 35% margin is a measurement on one program. Muon came out of people trying to make a tiny training program go faster [21]. For the margin to transfer, hidden matrix parameters have to dominate your step cost, and the Newton-Schulz iterations have to cost less than the steps they remove [2]; three of the named collaborators spent their contribution on reducing the cost of the orthogonalization itself [17]. Why it works is still open too. The original write-up says transformer updates often have high condition numbers, with a few directions dominating while others carry much smaller singular values [18], and the authors proposed that orthogonalization may let those lower-magnitude directions contribute more [19]. Shrijith Venkatramana, who wrote the dev.to explainer, wrote that the explanation "is an empirical hypothesis rather than the complete theoretical justification for Muon" [20].
What to watch
- Whether the PyTorch implementation ships a documented default split for embeddings, output heads, biases and norms.
- Whether anyone measures a fused QKV projection against Boza's separate Q, K and V finding on the same model.
- Published multi-billion-parameter runs that report wall-clock overhead for the Newton-Schulz iterations, not just step counts.