Transformers Can Learn Posterior Predictive Distributions In-Context

Transformers Can Learn Posterior Predictive Distributions In-Context, by Gyeonghun Kang, Changwoo J. Lee, and Xiang Cheng from Duke University, gives a constructive theory for prior-data fitted networks, or PFNs, showing that a transformer can approximate the full Gaussian process posterior predictive distribution rather than only a point estimate. The key result is an explicit attention-based algorithm: masked self-attention implements a Richardson-style iterative solver for the posterior predictive mean and variance, and a shallow MLP converts those moments into binned probabilities via a softmax over discretized density bins. The approximation error is bounded by three terms: exponential decay in attention depth L, discretization error of order 1/C from the number of bins, and tail truncation outside the support interval. The paper then identifies why PFNs generalize to context sizes n beyond pretraining: without normalization, the admissible step size shrinks like 1/n for both linear and RBF kernels, causing instability or under-convergence, while row-normalized attention acts as Jacobi preconditioning and stabilizes the spectrum. Yet even with normalization, the condition number still grows as Θ(n) for RBF kernels, so deeper stacks are required; the authors prove that to reach fixed accuracy, depth must scale at least on the order of n log(1/ε). Experiments on Bayesian linear regression, RBF regression, and real Sacramento housing and Walker Lake spatial data confirm the theory: increasing depth and bin resolution reduces total variation error, and normalized models maintain calibrated 90% and 95% intervals much farther outside the pretraining range.