Clustered Attractor Manifolds and Dynamical Condensation in Self-Attention

This paper analyzes a minimal normalized self-attention system as a dynamical process in which token similarities create attention weights and those weights reshape the representations. In the scaling regime d=N, the central quantity is the overlap gap: when tokens are more similar to members of their own cluster than to any competing cluster by a finite amount, inter-cluster attention is exponentially suppressed as dimension grows. The result is a high-dimensional manifold of locally attracting clustered fixed points, spanning a few macroscopic groups to an extensive number of microscopic fragments; internal distortions contract, while collective cluster rotations remain neutral. Starting from normalized Gaussian vectors, the dynamics exhibit a finite-sharpness attention-condensation transition: low beta produces diffuse attention and rank collapse, intermediate beta yields a macroscopic cluster coexisting with condensed microscopic groups, and higher beta produces broadly distributed microscopic fragmentation. For a nearly rank-collapsed parent cluster, the control parameter is effective sharpness alpha=beta epsilon^2; at beta=0.8, simulations show broad collective alignment followed by tight-cluster formation. At finite N, weak leakage causes eventual coarsening, but fragmented-state lifetimes grow exponentially with beta sqrt(N), making the order of limits essential. Unlike static random-energy-model softmax, which requires beta scaling as log N, feedback dynamically generates finite overlap gaps and enables condensation at beta=O(1). The analysis also explains why linear attention does not condense through the same mechanism and suggests implications for architectures such as ChatGPT and other transformers; the manuscript notes that OpenAI’s ChatGPT, specifically GPT-5.5, assisted with drafting and exploratory calculations.