Nonparametric In-Context Learning under Growing Geometric Complexity: Minimax Optimality and Local Geometry-Adaptivity of Transformers

This paper asks whether in-context learning can remain statistically optimal when data lie on a growing mixture of locally different manifolds rather than a single Euclidean space. It models components with varying intrinsic dimensions, Hölder smoothness, sampling masses, separation, reach, and bounded covariate noise, then derives an aggregate minimax risk that preserves each component’s effective sample size. The proposed estimator first fits a higher-order local tangent graph, projects observations into intrinsic coordinates, and applies chartwise local-polynomial regression. A structure-informed softmax transformer with ReLU feed-forward networks compiles this geometry-first procedure into one forward pass, achieving approximation error smaller than any fixed inverse polynomial, logarithmic depth, and polynomial resource growth. The theory also separates in-context error, transformer approximation, finite-task meta-training, and optimization error, showing that near empirical-risk minimization reaches the same aggregate rate with sufficient training tasks. This provides a mathematical account of the kind of few-shot adaptation associated with ChatGPT and other large language models, without analyzing a particular commercial model. In a synthetic benchmark with ambient dimension 64, context size 40000, and 12 heterogeneous components, an empirical softmax workspace predictor trained on 1000000 prompts achieved test MSE 0.505 in units of 10^-3, 26.75% lower than the geometry-informed LP-CV benchmark; the model used 286401 trainable parameters.