Small-Scale Experiments: Are We There Yet?

This paper argues that scaling laws do exist far below production-model scale, but small models are difficult to tune, so the laws remain hidden unless researchers reach the fully optimized hyperparameter frontier. Experiments with a Llama decoder in Meta Lingua, using OpenAI’s p50k_base tokenizer, the 100B-token FineWeb-Edu subset, and NVIDIA A100 GPUs, span 4M to 268M effective parameters. Random-search studies show that reliable scaling requires far more tuning than typical practice: the relationship is absent with only 4 or 16 configurations, becomes visible at 64, and is accurate at 256 configurations. Among methodological choices, learning-rate decay reduced held-out test mean-squared error by 98%, while parameter-count conventions and tied scaling exponents mattered much less. The proposed noisy quadratic limit reveals that hyperparameter loss surfaces become lower-dimensional as models grow, with the effective hyperparameter count falling to 1, explaining why large models are easier to tune. The authors combine this diagnostic with scaling laws and perplexity-capability correspondence, validating conclusions on AI2 ARC, BoolQ, MMLU, OpenBookQA, and PIQA while warning that extrapolation eventually becomes statistically unreliable. In a transformer normalization case study, small-scale experiments recover the established result that pre-normalization scales better than post-normalization: post-norm remains more hyperparameter-sensitive, whereas pre-norm yields a cleaner scaling law and stronger compute efficiency near observed data. The practical recommendation is to explore hundreds of configurations at small scale, verify diagnostics, and transfer the resulting design upward rather than relying on naïve extrapolation.