How Many Different Outputs Can a Transformer Generate?

This paper argues that a transformer’s output diversity is sharply limited by geometry and finite numerical precision, not just by training or scale. The authors define an accessible sequence as one that can be forced by some prompt under greedy decoding, then prove that every transformer generates only a finite set of such sequences: with prompt length m, the number of accessible outputs is bounded by the packing number of the bounded embedding support, yielding a maximal accessible length that grows only linearly with m. Under a worst-case ball enclosure, the critical slope is at most d·ln(1+2r/ε)/ln|V|, and beyond that threshold the fraction of accessible length-n sequences decays exponentially as O((1/|V|)^n). They extend the argument to arbitrary prompt lengths using a mean-field transformer formulation and Wasserstein precision, obtaining a comparable finite-output bound even with unbounded context and computation. Empirically, they test the theory with the cramming task on Pythia, Qwen-2.5, Llama-3.2, and Gemma-3 models: accessibility stays high until a sharp length cutoff, then collapses, and the 50% accessibility length n50 scales almost perfectly linearly with memory size m, with R² values of 0.999 on PG19 and 0.995 on random sequences. The theoretical slope is within a factor of about 5 to 20 of the measured slope, and this gap shrinks to roughly 4.6–10.8 when they replace the spherical-support assumption with ellipsoid or cone enclosures and non-uniform decoder-cell volumes. A copying experiment shows the same abrupt transition, with sigmoid fits achieving median R² = 0.95 across models.