NTH

The Role of Quantization in Loss Spikes

AuthorsZaiqian Chen, Aditya Somasundaram, Sophie Li, Sean McLeish, John Kirchenbauer, Tom Goldstein, Micah Goldblum

November 8, 2026 2 min read
Watch on YouTube
The one-line take

The authors link delayed language-model training spikes to growing quantization error and show that using momentum SGD on the output head can help keep training stable.

Key results

28M
Precision-test model size

Parameter count for models used in the LM-head precision experiments.

16B
Precision-test training tokens

Training horizon for the BF16 and FP32 comparison.

28
Chinchilla-optimal horizon multiple

The 16B-token run was roughly this many times the compute-optimal token horizon.

0.99
LM-head SGD momentum

Momentum coefficient used for the targeted SGD mitigation.

What the paper found

The paper identifies a hidden numerical cause of delayed, unrecoverable “time-bomb” loss spikes in language-model pretraining. In modernized GPT-2 models trained on FineWeb, AdamW’s coordinate-wise updates can break the language-model head’s zero-sum gradient structure, causing the mean output logit to drift. That shift is harmless in exact arithmetic, but at BF16 precision it increases rounding error until meaningful differences between logits—and therefore the predicted token distribution—are corrupted. In tests, 28M-parameter models trained for 16B tokens, roughly 28 times the Chinchilla-optimal horizon, failed earliest with BF16 true precision, later with BF16 mixed precision, while FP32 remained stable; the comparison covered five seeds. The proposed fix is targeted: use SGD with momentum of 0.99 for the language-model head while keeping AdamW for the rest of the model. This suppresses logit growth and quantization error; z-loss and µ-centering also keep the error bounded, pointing to a shared stabilization mechanism. The experiments include GPT-2-style models and runs on NVIDIA B200 GPUs. The paper also discloses using ChatGPT, Claude, and Gemini for brainstorming, coding, and editing.

Original abstract

Loss spikes in language model pretraining can waste substantial compute and derail otherwise healthy training runs. A primary driver of these instabilities is the growth and divergence of output logits in the final language modeling (LM) head. In this work, we study a particularly costly form of this instability associated with output logit growth: delayed, terminal divergences that emerge only after prolonged training, which we call time-bomb spikes. We find that these instabilities are preceded by rapidly growing quantization error in the LM head, which eventually culminates in a loss spike. Crucially, reducing numerical precision accelerates time-bomb spikes, while increasing precision improves stability. We then propose a simple yet effective mitigation that replaces AdamW on the LM head with SGD with momentum, suppressing logit growth and improving stability over long training horizons. Finally, we find that existing mitigation techniques from the literature (along with our own) all reduce quantization error, suggesting a shared numerical mechanism behind their effectiveness. Together, our results identify finite precision error as a hidden driver of time-bomb spikes and show that suppressing this error is key to stable long-horizon pretraining.

Read the original paper

More in Optimization

Browse all 42 papers →