Unified Neural Scaling Laws

Unified Neural Scaling Laws, from Mila researchers Ethan Caballero, David Krueger, and Irina Rish with Priyank Jaini of Google DeepMind, proposes a single multivariate curve family that models deep-network performance as model size, dataset size, training steps, inference steps, batch size, and hyperparameters vary together. The key novelty is a nested functional form built from multivariate broken neural scaling laws with smooth “hyperbreaks,” plus explicit terms for bottlenecks, irreducible performance limits, overfitting, and nonmonotonic hyperparameter effects such as learning rate and initialization scale. Unlike earlier scaling laws such as the Kaplan and Hoffmann Chinchilla-style formulas, or Muennighoff et al.’s data-constrained language model law, UNSL can represent transitions that reverse direction and does so with the same asymptotic expressivity as simpler baselines. On held-out extrapolation, it is substantially more accurate: in downstream image classification it is the best model on 60.87% of tasks, versus 21.74% for the next best form, and in language it wins on 88.89% of tasks. Reported RMSLE drops include ImageNet bivariate extrapolation from 2.54e-1 for DC to 8.57e-3 for UNSL, and language bivariate extrapolation from 6.24e-2 for DC to 7.82e-3 for UNSL. The paper also shows the law extrapolating reinforcement learning, test-time chain-of-thought scaling, width-depth tradeoffs, and sparse parity, while revealing a practical limit: accurate forecasting past a regime change requires training data near the relevant hyperbreak.