MuonGS: Pretraining LLMs with Relaxed Weight-Norm Control
AuthorsAthanasios Glentis, Dawei Li, Mingyi Hong
AffiliationsUniversity of Minnesota
MuonGS aims to make LLM pretraining more efficient by controlling how weights grow without fixing their norm.
Key results
Maximum estimated training-token saving over MuonH at matched validation loss.
Smaller GPT-style model tested on FineWeb.
Larger GPT-style model tested on FineWeb.
Active non-embedding parameters in the Ultra-FineWeb mixture-of-experts model.
Total parameters in the Ultra-FineWeb mixture-of-experts model.
What the paper found
MuonGS addresses a key problem in Muon pretraining: as weight norms grow, each update becomes smaller relative to the weights, limiting how far optimization can explore. The paper measures this through accumulated effective learning rate and shows that vanilla Muon can accumulate only logarithmically in the worst case. MuonGS instead applies a Gram–Schmidt projection to remove each update’s radial component, allowing the weight norm to grow without fixing it to a radius. The analysis proves at most O(√t) norm growth and at least Ω(√T) accumulation, including over later training windows; for smooth scale-invariant objectives, it also establishes an O(T^-1/2) stationarity guarantee. On FineWeb, MuonGS achieves lower validation perplexity than MuonH at longer horizons, with an estimated token-equivalent speedup of up to 15% at matched loss. Tests cover dense GPT-style models with 160M and 400M parameters, plus an Ultra-FineWeb mixture-of-experts model with 135M active and 1B total parameters. The long-horizon motivation is relevant to contemporary systems such as Meta’s Llama 3 and Alibaba’s Qwen3, although they are not evaluated here.
Original abstract
Weight decay and explicit weight-norm constraints improve optimizer performance in language model pretraining, yet a unified understanding of why such control helps and how much is needed remains incomplete. We analyze norm control in Muon over the entire training trajectory using the \emph{accumulated effective learning rate}, $C_T=\sum_{t<T}η_t/\|W_t\|_F$, where $η_t$ is the nominal learning rate and $η_t/\|W_t\|_F$ is the effective learning rate relative to the weight norm. For bounded update directions, accumulation over any training interval bounds relative weight displacement. We show that vanilla Muon can accumulate only logarithmically, potentially limiting exploration and helping explain its slower empirical progress, whereas MuonW and MuonH guarantee at least linear accumulation under the stated conditions. Going beyond bounded-norm control, we show that sufficiently slow norm growth guarantees faster accumulation than the logarithmic worst case, opening up less restrictive optimizer designs. Based on this insight, we propose MuonGS, which uses Gram--Schmidt (GS) projection to remove the radial component of each Muon update without fixing the weight norm. With bounded update directions and a constant nominal learning rate, we prove that the weight norm grows at most proportionally to $\sqrt{t}$ after $t$ steps. Consequently, MuonGS guarantees $C_T=Ω(\sqrt{T})$, with accumulation sustained even in later training. Further, for smooth scale-invariant objectives we establish its $O(T^{-1/2})$ stationarity guarantee. In pretraining experiments with 160M- and 400M-parameter dense models as well as a 135M active- and 1B total- parameter MoE model, MuonGS achieves lower validation perplexity than MuonH at long training horizons. In particular, for the dense model it achieves an estimated token-equivalent speedup up to about $15\%$ at matched validation loss to MuonH.
Read the original paperMore in Optimization
Browse all 42 papers →The Role of Quantization in Loss Spikes
Zaiqian Chen, Aditya Somasundaram, Sophie Li, Sean McLeish, John Kirchenbauer, Tom Goldstein, Micah Goldblum
The authors link delayed language-model training spikes to growing quantization error and show that using momentum SGD on the output head can help keep training stable.
Your LR Schedule May Not Be Principled: Effective Learning Rate Scheduling in Weight-Normalized Training
Qifan Yu, Xinyu Ma, Xingyan Bin, Di He
A new learning-rate schedule targets the effective step size and improves large-scale pretraining over cosine decay.
A provable quantum advantage for approximate optimization via decoded quantum interferometry
Maximilian J. Kramer, Elies Gil-Fuster, Benjamin D. M. Jones, Jens Eisert, Franz J. Schreiber
This paper proves that, for a specially constructed optimization problem, quantum interferometry can achieve approximation scores that classical algorithms cannot match efficiently.