Your LR Schedule May Not Be Principled: Effective Learning Rate Scheduling in Weight-Normalized Training
AuthorsQifan Yu, Xinyu Ma, Xingyan Bin, Di He
AffiliationsByteDance Seed · Peking University
A new learning-rate schedule targets the effective step size and improves large-scale pretraining over cosine decay.
Key results
Token budget for the primary Mixture-of-Experts schedule comparison.
Approximate AIR improvement over cosine at matched cumulative learning rate.
Token budget for the larger-scale Mixture-of-Experts run.
Larger-scale downstream benchmark average; cosine scores 56.22.
Approximate share of the AIR–Cycloid final-loss gap transferred in the controlled comparison.
What the paper found
In experiments involving ByteDance, this study argues that weight-normalized training’s advantage over weight decay comes mainly from its effective learning rate (ELR)—the nominal learning rate divided by parameter-block norm—not from normalization alone. When ELR trajectories are matched, the two methods’ training-loss dynamics largely align. The authors derive an inverse-square-root decay from a simplified directional-alignment model and turn it into AIR, an affine schedule that meets chosen peak and terminal learning rates. In Mixture-of-Experts pretraining on 128B tokens, AIR cuts terminal loss by about 0.01 versus cosine decay at matched cumulative learning rate. The gains are concentrated late in training: swapping the late schedule transfers about 90% of the loss gap in a controlled comparison. At larger scale, a run trained on 500B tokens achieves a downstream benchmark average of 57.78 with AIR, compared with 56.22 for cosine. The results suggest that learning-rate design should target ELR trajectories, especially their late-stage behavior, rather than nominal learning rates alone.
Original abstract
Weight-normalized (WN) training is emerging as a compelling alternative to conventional weight decay (WD), with recent studies reporting substantial pretraining loss gains. We show that these gains are driven primarily by the induced effective learning rate (ELR) trajectory rather than by normalization alone. Matching ELR trajectories largely collapses the training loss dynamics of WN and WD. Schedule design should therefore target ELR rather than the nominal learning rate (LR) alone. Building on this principle, we propose AIR (Affine Inverse-square-Root decay), which combines a theoretically derived $1/\sqrt{t}$ decay with an affine adjustment to satisfy prescribed peak and terminal LRs over a finite training horizon. We further identify late-stage ELR scheduling as a key source of its gains, because a shared late schedule attenuates the effect of earlier LR choices. Experiments on 128 B-token Mixture-of-Experts pretraining demonstrate that AIR outperforms cosine and other candidate schedules in training loss, downstream performance, and hyperparameter robustness, reducing terminal training loss by approximately 0.01 relative to cosine at matched cumulative LR. Applying the same ELR trajectory to WD reproduces the corresponding training dynamics, showing that the benefits extend beyond fixed-norm training and establishing ELR as the principled scheduling object.
Read the original paperMore in Optimization
Browse all 42 papers →MuonGS: Pretraining LLMs with Relaxed Weight-Norm Control
Athanasios Glentis, Dawei Li, Mingyi Hong
MuonGS aims to make LLM pretraining more efficient by controlling how weights grow without fixing their norm.
The Role of Quantization in Loss Spikes
Zaiqian Chen, Aditya Somasundaram, Sophie Li, Sean McLeish, John Kirchenbauer, Tom Goldstein, Micah Goldblum
The authors link delayed language-model training spikes to growing quantization error and show that using momentum SGD on the output head can help keep training stable.
A provable quantum advantage for approximate optimization via decoded quantum interferometry
Maximilian J. Kramer, Elies Gil-Fuster, Benjamin D. M. Jones, Jens Eisert, Franz J. Schreiber
This paper proves that, for a specially constructed optimization problem, quantum interferometry can achieve approximation scores that classical algorithms cannot match efficiently.