Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention

This paper, from Stanford University, the Kempner Institute at Harvard, MIT, and Anthropic, argues that larger language models learn more not just because they are more expressive, but because scaling changes optimization dynamics on long-tailed task mixtures. Using a phenomenological reading of Chinchilla-style scaling, the authors distinguish tasks that smaller models can eventually recover with more data from tasks that require model scaling even with asymptotically unlimited data. In a synthetic mixture of orthogonal linear regression tasks, they show that width selects features by utility, defined as frequency times spectral magnitude, so small models spend capacity on high-frequency, low-complexity modes and fail on rare or complex ones. They then identify the key mechanism as reduced gradient interference: once a larger model has enough residual capacity to fit common tasks, gradients from those tasks shrink, leaving rare-task features less likely to be overwritten between sparse observations. In matched-frequency injection experiments, larger models retain rare-task signal across gaps, while narrow models exhibit an update-and-forget loop. The authors validate the same pattern in OLMo pretraining from 4M to 4B parameters on Dolma v1.7 by injecting two novel tasks, comparison and modular addition, at controlled frequencies. Only the larger OLMo models learn the lowest-frequency tasks, encode more task-specific features in their representations via distributed alignment search and Fourier analysis, and show near-orthogonal non-task gradients; for the 1B model, non-task gradient cosine similarity with the task direction is about 7.58×10^-5 ± 0.02, versus 0.10 ± 0.09 for the 20M model. The central claim is that memorization-like retention can be a prerequisite for learning rare, generalizable structure.