Correcting Stochastic Update Bias in Preconditioned Language Model Optimizers

This paper shows that stochastic preconditioned language-model optimizers such as AdamW, Sophia-G, and Shampoo are biased estimators of the ideal population update because the gradient and preconditioner are usually computed from the same minibatch, and because matrix or scalar inversion is nonlinear. The authors separate these effects into gradient–preconditioner coupling bias and inverse-preconditioner bias, then correct both with a single-batch framework: cross-fitting estimates the numerator gradient and denominator preconditioner from independent microbatch groups, and a delta-method variance correction subtracts the leading inverse bias using microbatch variability. For AdamW and Sophia, the correction is applied elementwise to diagonal preconditioners; for Shampoo, it is applied in the eigenspace of the averaged matrix preconditioner. On Qwen2.5-0.5B pretraining over 256K packed FineWeb-Edu sequences, the corrected optimizers reduce held-out cross-entropy by 0.1489 nats for AdamW using a leave-one-out plus Jensen variant, 0.0701 nats for Sophia-G, and 0.1103 nats for Shampoo. In a mixed-quality pretraining setup with 20 percent span-replaced training sequences, AdamW still improves by 0.0191 nats on clean held-out data despite higher training loss, suggesting reduced overfitting to corrupted spans. Instruction tuning on 32K Alpaca-style examples is mostly neutral, with small gains for retuned AdamW and near-ties for Sophia and Shampoo. The main novelty is not a new optimizer geometry, but a statistical bias-correction layer that makes existing preconditioned updates closer to their population target.