NTH

Your LR Schedule May Not Be Principled: Effective Learning Rate Scheduling in Weight-Normalized Training

AuthorsQifan Yu, Xinyu Ma, Xingyan Bin, Di He

AffiliationsByteDance Seed · Peking University

November 8, 2026 2 min read
Watch on YouTube
The one-line take

A new learning-rate schedule targets the effective step size and improves large-scale pretraining over cosine decay.

Key results

128B
Pretraining tokens

Token budget for the primary Mixture-of-Experts schedule comparison.

0.01
Terminal loss reduction

Approximate AIR improvement over cosine at matched cumulative learning rate.

500B
Large-scale pretraining tokens

Token budget for the larger-scale Mixture-of-Experts run.

57.78
AIR downstream average

Larger-scale downstream benchmark average; cosine scores 56.22.

90%
Loss gap transferred by late-stage swap

Approximate share of the AIR–Cycloid final-loss gap transferred in the controlled comparison.

What the paper found

In experiments involving ByteDance, this study argues that weight-normalized training’s advantage over weight decay comes mainly from its effective learning rate (ELR)—the nominal learning rate divided by parameter-block norm—not from normalization alone. When ELR trajectories are matched, the two methods’ training-loss dynamics largely align. The authors derive an inverse-square-root decay from a simplified directional-alignment model and turn it into AIR, an affine schedule that meets chosen peak and terminal learning rates. In Mixture-of-Experts pretraining on 128B tokens, AIR cuts terminal loss by about 0.01 versus cosine decay at matched cumulative learning rate. The gains are concentrated late in training: swapping the late schedule transfers about 90% of the loss gap in a controlled comparison. At larger scale, a run trained on 500B tokens achieves a downstream benchmark average of 57.78 with AIR, compared with 56.22 for cosine. The results suggest that learning-rate design should target ELR trajectories, especially their late-stage behavior, rather than nominal learning rates alone.

Original abstract

Weight-normalized (WN) training is emerging as a compelling alternative to conventional weight decay (WD), with recent studies reporting substantial pretraining loss gains. We show that these gains are driven primarily by the induced effective learning rate (ELR) trajectory rather than by normalization alone. Matching ELR trajectories largely collapses the training loss dynamics of WN and WD. Schedule design should therefore target ELR rather than the nominal learning rate (LR) alone. Building on this principle, we propose AIR (Affine Inverse-square-Root decay), which combines a theoretically derived $1/\sqrt{t}$ decay with an affine adjustment to satisfy prescribed peak and terminal LRs over a finite training horizon. We further identify late-stage ELR scheduling as a key source of its gains, because a shared late schedule attenuates the effect of earlier LR choices. Experiments on 128 B-token Mixture-of-Experts pretraining demonstrate that AIR outperforms cosine and other candidate schedules in training loss, downstream performance, and hyperparameter robustness, reducing terminal training loss by approximately 0.01 relative to cosine at matched cumulative LR. Applying the same ELR trajectory to WD reproduces the corresponding training dynamics, showing that the benefits extend beyond fixed-norm training and establishing ELR as the principled scheduling object.

Read the original paper

More in Optimization

Browse all 42 papers →
02Optimizer

The Role of Quantization in Loss Spikes

Zaiqian Chen, Aditya Somasundaram, Sophie Li, Sean McLeish, John Kirchenbauer, Tom Goldstein, Micah Goldblum

The authors link delayed language-model training spikes to growing quantization error and show that using momentum SGD on the output head can help keep training stable.

Read analysis