build1 publisher
Muon throws away a gradient's singular values and keeps its directions
The optimizer behind an October 2024 NanoGPT speed record is now part of the mainstream PyTorch stack. Its scope covers hidden matrix parameters, so the rest of the model still needs a second optimizer.
Publishers:dev.to
Reality
- Evidence52
- Adoption32
- Hype gap+12
- Incentives40
- Confidence45