Meta Models: Models that Model Learning
AuthorsPeihang Li, Ziming Liu
AffiliationsMetaCircle The University of Hong Kong University of California, Berkeley · Shanghai Qizhi Institute Tsinghua University*Corresponding author
Instead of running a model-training experiment first, the authors use prior runs to predict how the next one will learn.
Key results
Mean within-group Spearman correlation across held-out conditions; XGBoost scores 0.875.
Reduction after adding the first 20% of each target trajectory to context.
Improvement using KAN-learned configuration coordinates.
Percentage points; XGBoost with the same historical-checkpoint context scores 3.40.
Across 14 later checkpoints, versus 0.0576 for the official scaling law.
What the paper found
This paper treats language-model training records as numeric data that a reusable meta-model can learn from, rather than as text for an LLM to interpret. With a frozen TabPFN backbone and historical runs supplied as context, the same model predicts final loss, training curves, optimizer and data-mixture effects, internal training metrics, and downstream accuracy—without task-specific weight updates. On Delphi’s optimizer sweep, TabPFN reaches a mean within-group Spearman correlation of 0.967, compared with 0.875 for XGBoost. Feeding in the first 20% of a target run’s observations reduces future-loss error by 56%, and adding scale-aware coordinates learned by Kolmogorov–Arnold Networks cuts StepLaw’s out-of-distribution configuration-selection regret by 80%. For downstream accuracy on larger DataDecide models, its macro-accuracy error is 1.62 percentage points, versus 3.40 for XGBoost. The paper also tests a forecast on the public 535B-parameter Marin Hero Run: using early observations from that run, TabPFN’s mean absolute error on 14 later checkpoints is 0.0216, compared with 0.0576 for the official scaling law. The results suggest that numeric in-context learning can support diverse training questions with little adaptation, while reliable extrapolation and demonstrated compute savings remain open challenges. The paper separately discloses using OpenAI ChatGPT and Anthropic Claude as research and coding assistants.
Original abstract
Every neural-network training run produces not only a model, but also evidence about how models learn. Can these accumulated experiments enable us to predict and compare new training runs before executing them? We study meta-models: models that predict the learning behavior of other models. From this perspective, final-performance prediction, learning-curve prediction, and hyperparameter selection are different queries to the same training response function. We take a first, humble empirical step toward this goal: treating real language-model training records as native numeric data, we find that an off-the-shelf tabular in-context learner, TabPFN, can match or outperform a fine-tuned LLM meta-model in multiple evaluated settings. Whereas recent work fine-tunes LLMs on tokenized configurations at a cost of tens to hundreds of GPU-hours, TabPFN requires seconds of fit-and-predict time without task-specific weight updates. We argue that meta-modeling is better treated as its own numeric modality rather than as a text task for LLMs. Going one step further, augmenting TabPFN with Kolmogorov-Arnold Networks (KANs) further improves accuracy on selected prediction tasks. Finally, we pre-registered a forecast of the Marin Hero Run, a 535B-parameter public training run, whose ground truth is revealed point by point as training progresses, putting meta-model predictions to the test against training outcomes not yet observed at the time of forecasting.
Read the original paperMore in Neural Networks
Browse all 28 papers →Curriculum unlearning in humans and neural networks
Jirko Rubruck, Alice Zhang, Andrew Saxe, Christopher Summerfield
Simple-to-complex curricula may work by helping humans and neural networks unlearn overly simple rules and build richer representations.
Do vision models know that three is three?
Ankit Sonthalia, Arnas Uselis, Seong Joon Oh
Vision models often learn a concept change clearly within one visual setting, but may represent that same change differently in another.
Continuous Memory Machines
Ciaran Regan, Kai Arulkumaran, Luke Darlow, Stefania Druga, Sebastian Risi, Llion Jones
Continuous Memory Machines give recurrent networks separate short- and long-term memories that can read from and reorganize one another.