Transformers as In-Context Samplers: From Closed-Form Diffusion to Estimation-Free Sampling

This paper expands in-context learning from prediction to generation by asking whether a frozen transformer can act as a sampler: given independent samples from an unknown distribution, can it produce a new sample without parameter updates or an estimated score network? The theory answers yes. Standard softmax attention computes Gaussian-mixture responsibility weights and their weighted empirical mean, while feedforward layers perform Euler updates, allowing a transformer to implement closed-form diffusion exactly, including a smoothed variant that reduces memorization. A second theorem shows approximation of a finite Estimation-Free Sampling procedure, combining particle gradient descent with reverse-time transport, so transformers can execute both score-based and estimation-free samplers in context. Experiments with a GPT-2-style model show compositional out-of-distribution generation: training excludes a smile-shaped component, yet smile points in the prompt lead to new samples along that unseen curve. In a 16-layer, 32-dimensional model trained on two-moons and related point clouds, hidden states first spread toward a more uniform geometry and later recover the prompt’s structured distribution. The same U-shaped layerwise pattern appears in normalized embeddings from OpenAI’s GPT-2, Meta’s Llama-3.3-70B-Instruct, Cerebras-GPT, and Qwen2.5, measured with RBF MMD2 and an interacting-particle energy. However, the authors emphasize an expressivity-versus-mechanism gap: these theorems prove that transformers can implement the algorithms, not that pretrained models literally do so, and smaller Qwen2.5 variants show a weaker intermediate transport phase.