NTH

Interpretable weight-space programming builds transparent transformers and brain encoding models

AuthorsYash Mehta, Chandan Singh, Weijia Xu, Richard J. Antonello, Adeen Flinker, Nima Mesgarani, Michael F. Bonner, Nebojsa Jojic, Jianfeng Gao

AffiliationsMicrosoft Research · Johns Hopkins University · Columbia University · New York University ‡ github.com/csinva/weight-space-programming-autoresearch

November 19, 2026 2 min read
Watch on YouTube
The one-line take

An LLM agent hand-builds interpretable neural networks that predict brain activity and suggest what computations the brain may be using.

Key results

100%
Algorithmic hidden accuracy

Claude Opus 5.5 achieved this on its algorithmic task runs.

166
Median parameters at perfect visible accuracy

Smallest algorithmic models found by Claude Opus 5.5.

0.443
NSD early visual cortex correlation

iWSP prediction on the Natural Scenes Dataset; best interpretable baseline scored 0.400.

0.317
NSD ventral visual cortex correlation

iWSP prediction on the Natural Scenes Dataset; best interpretable baseline scored 0.256.

0.97
Scan-position signal correlation

How closely story-position features tracked the scan-timing response pattern.

0.0052
ECoG gain from combining iWSP with Whisper

Increase in hidden-set correlation over Whisper alone, from 0.070 to 0.075.

What the paper found

Interpretable weight-space programming, or iWSP, asks an LLM agent to write neural-network weights and features directly in code, then revise them using evaluation scores—without backpropagation or training. Separate audits and tests that rebuild models from short descriptions check whether the resulting computations are genuinely interpretable. Using Anthropic’s Claude Opus 5.5 and Gemini 4 Carbon, the method produced transformers that reached 100% hidden accuracy on algorithmic tasks; the median smallest model reaching perfect visible accuracy used 166 parameters. For visual brain encoding, its 14 explicit computations outperformed interpretable baselines on the Natural Scenes Dataset, reaching correlations of 0.443 in early visual cortex and 0.317 in ventral visual cortex. In story-listening fMRI, iWSP exposed scan-position features that tracked an unintended timing signal at 0.97 correlation, pointing to the need to trim scan edges. For auditory ECoG, its readable acoustic features complemented Whisper: combining them raised hidden-set correlation from 0.070 to 0.075, a gain of 0.0052. The paper also compares language-fMRI predictions with GPT-2 XL. Overall, iWSP’s contribution is a practical way to search for predictive models whose internal computations can be inspected, tested, and reconstructed—while its results still rely on known computational motifs rather than new ones.

Original abstract

Neural networks are powerful predictors, yet their opacity limits their usefulness in science, where the goal is explanation rather than mere prediction. In neuroscience, for example, deep encoding models accurately predict brain activity in response to natural stimuli, but struggle to reveal which stimulus features drive those predictions. Here we introduce interpretable weight-space programming (iWSP), a method in which an LLM agent iteratively builds interpretable models by explicitly programming their weights and features in code, without gradient-based training (backpropagation). Without backpropagation to tune parameters, the agent must draw on its prior knowledge and reason over previous trials to hypothesize, hand-craft, and refine human-interpretable computations mirroring the workflow of a human scientist. LLM audits confirm the interpretability of the hand-crafted models. We first validate iWSP by tasking it with constructing transformers for canonical benchmarks such as arithmetic and sentiment classification, finding that LLM agents can successfully craft compact, accurate networks directly. Next, we apply iWSP to build neural encoding models across vision, language, and audition. In the visual cortex, iWSP outpredicts existing interpretable baselines, uncovers distinct computations across early/higher visual areas, and generalizes to unseen data. In language fMRI, iWSP surfaces a confound in the data and in auditory ECoG, it systematically discovers interpretable acoustic features that predict neural activity. Together, these results establish iWSP as a general paradigm for aligning high predictive performance with human-interpretable scientific discovery.

Read the original paper

More in AI for Science

Browse all 58 papers →