Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs

This ICLR 2026 paper from UW-Madison and Amazon Web Services extends Chinchilla-style scaling laws to include model architecture, targeting inference-efficient LLMs rather than compute-optimal training alone. The authors show that, at fixed parameter budgets, hidden size, the MLP-to-attention ratio, and grouped-query attention all materially affect throughput: larger hidden sizes, higher MLP-to-attention ratios, and higher GQA consistently increase tokens per second on both LLaMA-3.2 and Qwen3 variants. Using more than 200 decoder-only models trained from 80M to 3B parameters and 8B to 100B tokens, they fit a two-step conditional scaling law that first estimates the optimal loss under Chinchilla and then calibrates architectural effects through simple multiplicative or additive terms. The resulting predictors are accurate across scales, with low MSE and strong rank correlation, and they reveal U-shaped loss curves for both hidden size and MLP-to-attention ratio, implying interior optima rather than monotonic scaling. At 1B and 3B scales, the method yields Panda-1B and Panda-3B architectures that improve average downstream accuracy by 2.1% and 0.6% over LLaMA-3.2 baselines, while the search-derived Surefire-1B and Surefire-3B variants preserve loss constraints and raise inference throughput by up to 42% on A100 with vLLM; the same efficiency gains reach 47% with SGLang on H200. The paper also finds that fitting the law with outlier MLP-to-attention ratios below 0.5 or above 5 hurts prediction quality, and that simple separable calibration outperforms more complex non-separable formulations.