Planning to Learn
AuthorsIan Osband
AffiliationsGoogle DeepMind
This paper makes classification more forward-looking by designing a loss that values not just immediate accuracy gains, but how much future learning an update enables.
Key results
Percentage-point improvement over cross-entropy.
Top-1 percentage-point improvement over cross-entropy.
Top-1 percentage-point improvement over cross-entropy.
Reduced from 1.467 under standard cross-entropy.
What the paper found
Planning to Learn treats training as a finite-budget allocation problem: each update should favor examples that can still improve before training ends. The paper introduces a horizon loss, defined by averaging future classification error over the remaining learning horizon, with that horizon estimated from the learning rates still to come. Its per-example gradient weight interpolates between exact policy gradient at zero horizon and cross-entropy at an effectively unlimited horizon. This lets the method shift from broad learning early to deprioritizing examples that are unlikely to become correct late in training. On ImageNet, it improves top-1 accuracy over cross-entropy by 1.0–2.4 percentage points across ResNet-50, ResNet-101, and ViT-S/16; on ResNet-50, the gain is 2.2 points. The advantage grows with label noise: the ResNet-50 gain rises from 3.3 points at 10% noise to 7.3 points at 50% noise. The paper also plans cross-entropy itself, lowering ResNet-50 test negative log-likelihood from 1.467 to 1.364. The framing connects finite-horizon allocation to policy-gradient LLM post-training, citing OpenAI’s reasoning work and DeepSeek-R1, although the experiments are in image classification.
Original abstract
Policy-gradient methods are central to modern reinforcement learning, including LLM post-training. When they struggle, the usual suspects are exploration, credit assignment and action-sampling noise. Classification has none of them. A classifier is a policy whose expected reward, its \emph{expected accuracy}, is the probability it assigns to the correct label, and because that label is known, the policy gradient is exact and smooth. Yet exact policy gradient loses to cross-entropy, even on expected accuracy. The exact gradient is myopic: it values an update only by what it buys now, but each update also sets where the next one starts, so an update's value depends on how much learning remains. Viewed this way, cross-entropy is patient accuracy, the total error an example would pay if its log-odds rose at unit speed forever, while exact policy gradient is the zero-horizon limit. Truncating this total at the learning that remains yields the horizon loss, a one-line change that moves from cross-entropy toward exact policy gradient as training runs out. In a simple allocation model, it provably escapes the trap that catches each endpoint. On MNIST and on ImageNet with ResNet-50, ResNet-101 and ViT-S/16, the horizon loss improves top-1 accuracy over cross-entropy at a flat learning rate, and the gain grows with label noise.
Read the original paperMore in Optimization
Browse all 39 papers →A provable quantum advantage for approximate optimization via decoded quantum interferometry
Maximilian J. Kramer, Elies Gil-Fuster, Benjamin D. M. Jones, Jens Eisert, Franz J. Schreiber
This paper proves that, for a specially constructed optimization problem, quantum interferometry can achieve approximation scores that classical algorithms cannot match efficiently.
AF-Muon: An AdamW-Free Muon Optimizer for Tied-Embedding Models
Arash Lagzian, Paniz Halvachi, Junming Zhang, Zhouhan Lin, Dianbo Liu
AF-Muon adapts Muon to tied embeddings, reducing optimizer memory while improving training across language, image, and protein models.
An $Ω(κ_y^8ε^{-6})$ Lower Bound for Stochastic NC-SC Bilevel Optimization with First-order Oracles
Zhihao Gu, Qilong Wu, Junchi Yang
This work proves that stochastic bilevel optimization fundamentally requires up to epsilon^{-6} oracle queries, showing existing methods are asymptotically optimal.