The Role of Quantization in Loss Spikes
AuthorsZaiqian Chen, Aditya Somasundaram, Sophie Li, Sean McLeish, John Kirchenbauer, Tom Goldstein, Micah Goldblum
The authors link delayed language-model training spikes to growing quantization error and show that using momentum SGD on the output head can help keep training stable.
Key results
Parameter count for models used in the LM-head precision experiments.
Training horizon for the BF16 and FP32 comparison.
The 16B-token run was roughly this many times the compute-optimal token horizon.
Momentum coefficient used for the targeted SGD mitigation.
What the paper found
The paper identifies a hidden numerical cause of delayed, unrecoverable “time-bomb” loss spikes in language-model pretraining. In modernized GPT-2 models trained on FineWeb, AdamW’s coordinate-wise updates can break the language-model head’s zero-sum gradient structure, causing the mean output logit to drift. That shift is harmless in exact arithmetic, but at BF16 precision it increases rounding error until meaningful differences between logits—and therefore the predicted token distribution—are corrupted. In tests, 28M-parameter models trained for 16B tokens, roughly 28 times the Chinchilla-optimal horizon, failed earliest with BF16 true precision, later with BF16 mixed precision, while FP32 remained stable; the comparison covered five seeds. The proposed fix is targeted: use SGD with momentum of 0.99 for the language-model head while keeping AdamW for the rest of the model. This suppresses logit growth and quantization error; z-loss and µ-centering also keep the error bounded, pointing to a shared stabilization mechanism. The experiments include GPT-2-style models and runs on NVIDIA B200 GPUs. The paper also discloses using ChatGPT, Claude, and Gemini for brainstorming, coding, and editing.
Original abstract
Loss spikes in language model pretraining can waste substantial compute and derail otherwise healthy training runs. A primary driver of these instabilities is the growth and divergence of output logits in the final language modeling (LM) head. In this work, we study a particularly costly form of this instability associated with output logit growth: delayed, terminal divergences that emerge only after prolonged training, which we call time-bomb spikes. We find that these instabilities are preceded by rapidly growing quantization error in the LM head, which eventually culminates in a loss spike. Crucially, reducing numerical precision accelerates time-bomb spikes, while increasing precision improves stability. We then propose a simple yet effective mitigation that replaces AdamW on the LM head with SGD with momentum, suppressing logit growth and improving stability over long training horizons. Finally, we find that existing mitigation techniques from the literature (along with our own) all reduce quantization error, suggesting a shared numerical mechanism behind their effectiveness. Together, our results identify finite precision error as a hidden driver of time-bomb spikes and show that suppressing this error is key to stable long-horizon pretraining.
Read the original paperMore in Optimization
Browse all 42 papers →MuonGS: Pretraining LLMs with Relaxed Weight-Norm Control
Athanasios Glentis, Dawei Li, Mingyi Hong
MuonGS aims to make LLM pretraining more efficient by controlling how weights grow without fixing their norm.
Your LR Schedule May Not Be Principled: Effective Learning Rate Scheduling in Weight-Normalized Training
Qifan Yu, Xinyu Ma, Xingyan Bin, Di He
A new learning-rate schedule targets the effective step size and improves large-scale pretraining over cosine decay.
A provable quantum advantage for approximate optimization via decoded quantum interferometry
Maximilian J. Kramer, Elies Gil-Fuster, Benjamin D. M. Jones, Jens Eisert, Franz J. Schreiber
This paper proves that, for a specially constructed optimization problem, quantum interferometry can achieve approximation scores that classical algorithms cannot match efficiently.