Large-Step Training Dynamics of a Two-Factor Linear Transformer Model

This paper derives an exact finite-step training model for a one-prompt linear self-attention transformer and shows that large learning rates can create new attractors rather than simply speeding convergence. After rescaling, gradient descent on the reduced one-prompt loss becomes the two-factor map Φµ(a,b) = (a − (ab − µ)b, b − (ab − µ)a), equivalent to unit-step gradient descent on ℓµ(a,b)=1/2(ab−µ)^2 with effective step size µ = 2η|yq|√κ. On the balanced slice a=b, the dynamics reduce to the cubic family studied in quadratic regression, giving sharp thresholds at µ = 2√2 − 2 ≈ 0.828 for monotone error contraction, µ = 1 for loss of stability and a flip bifurcation, and µ = 2 for divergence. The novel result is the full two-dimensional analysis: for 0 < µ < 2, the map has an explicit invariant Chebyshev ellipse Eµ, on which the error evolves as e↦e^3−3e, i.e. angle tripling e = 2 cos θ ⇒ e+ = 2 cos 3θ. This ellipse is not an attractor but a transversely repelling separatrix, while balanced invariant measures are transversely attracting only for 1 < µ < 2. The paper also proves that the strict interior contains no hidden off-balanced recurrent dynamics beyond algebraic landing sets. For mini-batch gradient descent, each batch induces its own Φν, so training is random switching between distinct separatrices; a single atypical batch can push the iterate across the full-batch boundary. The main implication is mechanistic: in linear-transformer training, increasing η can move optimization into catapult, periodic, chaotic, or divergent regimes, and minibatch noise can destabilize a population-stable solution by exposing modes with batch-specific effective µ past the thresholds.