Bridging Compute- and Data-Optimal Pretraining

Tian Qin, Kimia Hamidieh, and David Alvarez-Melis, from Harvard and MIT CSAIL, propose Compute-Data, or CD, scaling laws to extend Chinchilla-style compute-optimal scaling into the data-constrained regime. Their key variable is token effectiveness, η: a derived token from repetition or paraphrasing is converted into a fresh-token equivalent, with effective data defined as D plus ηD′. Using the OLMo3 architecture, the Dolma-3 150B corpus, and models from 14M to 600M parameters, they show that η is not constant: it decreases with model size, tokens-per-parameter ratio, and expansion amount, eventually saturating. The resulting power-law saturation model identifies three operational regimes—compute-bound, data-bound, and model-bound—and predicts a joint compute–data Pareto frontier rather than Chinchilla’s single allocation path. A fit using 14M and 30M models extrapolates to larger held-out models with RMSE 0.079 on log validation loss. Ablations show that a constant-saturation baseline has roughly 40% higher relative leave-one-out RMSE, confirming that dependence on model size and data availability is essential. Strategy recommendations are scale-dependent: paraphrasing is favored for models below 600M and limited data, while repetition becomes stronger as scale and tokens-per-parameter increase; extrapolation suggests four epochs are appropriate only around 3B-parameter models near Chinchilla data, and paraphrasing becomes ineffective at 7B or larger with data budgets of at least 4× Chinchilla.