NTH

Meta Models: Models that Model Learning

AuthorsPeihang Li, Ziming Liu

AffiliationsMetaCircle The University of Hong Kong University of California, Berkeley · Shanghai Qizhi Institute Tsinghua University*Corresponding author

November 7, 2026 2 min read
Watch on YouTube
The one-line take

Instead of running a model-training experiment first, the authors use prior runs to predict how the next one will learn.

Key results

0.967
Delphi optimizer ranking correlation

Mean within-group Spearman correlation across held-out conditions; XGBoost scores 0.875.

56%
Future-loss error reduction with run prefix

Reduction after adding the first 20% of each target trajectory to context.

80%
StepLaw OOD selection-regret reduction

Improvement using KAN-learned configuration coordinates.

1.62
DataDecide downstream macro-accuracy MAE

Percentage points; XGBoost with the same historical-checkpoint context scores 3.40.

0.0216
Marin Hero Run prospective forecast MAE

Across 14 later checkpoints, versus 0.0576 for the official scaling law.

What the paper found

This paper treats language-model training records as numeric data that a reusable meta-model can learn from, rather than as text for an LLM to interpret. With a frozen TabPFN backbone and historical runs supplied as context, the same model predicts final loss, training curves, optimizer and data-mixture effects, internal training metrics, and downstream accuracy—without task-specific weight updates. On Delphi’s optimizer sweep, TabPFN reaches a mean within-group Spearman correlation of 0.967, compared with 0.875 for XGBoost. Feeding in the first 20% of a target run’s observations reduces future-loss error by 56%, and adding scale-aware coordinates learned by Kolmogorov–Arnold Networks cuts StepLaw’s out-of-distribution configuration-selection regret by 80%. For downstream accuracy on larger DataDecide models, its macro-accuracy error is 1.62 percentage points, versus 3.40 for XGBoost. The paper also tests a forecast on the public 535B-parameter Marin Hero Run: using early observations from that run, TabPFN’s mean absolute error on 14 later checkpoints is 0.0216, compared with 0.0576 for the official scaling law. The results suggest that numeric in-context learning can support diverse training questions with little adaptation, while reliable extrapolation and demonstrated compute savings remain open challenges. The paper separately discloses using OpenAI ChatGPT and Anthropic Claude as research and coding assistants.

Original abstract

Every neural-network training run produces not only a model, but also evidence about how models learn. Can these accumulated experiments enable us to predict and compare new training runs before executing them? We study meta-models: models that predict the learning behavior of other models. From this perspective, final-performance prediction, learning-curve prediction, and hyperparameter selection are different queries to the same training response function. We take a first, humble empirical step toward this goal: treating real language-model training records as native numeric data, we find that an off-the-shelf tabular in-context learner, TabPFN, can match or outperform a fine-tuned LLM meta-model in multiple evaluated settings. Whereas recent work fine-tunes LLMs on tokenized configurations at a cost of tens to hundreds of GPU-hours, TabPFN requires seconds of fit-and-predict time without task-specific weight updates. We argue that meta-modeling is better treated as its own numeric modality rather than as a text task for LLMs. Going one step further, augmenting TabPFN with Kolmogorov-Arnold Networks (KANs) further improves accuracy on selected prediction tasks. Finally, we pre-registered a forecast of the Marin Hero Run, a 535B-parameter public training run, whose ground truth is revealed point by point as training progresses, putting meta-model predictions to the test against training outcomes not yet observed at the time of forecasting.

Read the original paper

More in Neural Networks

Browse all 28 papers →
01Neural Network

Curriculum unlearning in humans and neural networks

Jirko Rubruck, Alice Zhang, Andrew Saxe, Christopher Summerfield

Simple-to-complex curricula may work by helping humans and neural networks unlearn overly simple rules and build richer representations.

Read analysis
02Neural Network

Do vision models know that three is three?

Ankit Sonthalia, Arnas Uselis, Seong Joon Oh

Vision models often learn a concept change clearly within one visual setting, but may represent that same change differently in another.

Read analysis
03Neural Network

Continuous Memory Machines

Ciaran Regan, Kai Arulkumaran, Luke Darlow, Stefania Druga, Sebastian Risi, Llion Jones

Continuous Memory Machines give recurrent networks separate short- and long-term memories that can read from and reorganize one another.

Read analysis