Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It

This paper argues that pretrained transformers often stop reference-following long before their available depth is exhausted. Across 13 standard models, including Qwen3, Llama, OLMo, Gemma, and the 292B-MoE DeepSeek-V4-Flash, reliable reach is only 1.4 to 3.6 assignment lines, with a median of 2.2, and extra recurrent loops in Ouro and Huginn add little. The proposed fix is a rank-8 residual-stream LoRA inserted at a model-specific early or middle layer while freezing every original weight. In Qwen3-8B, the adapter trains only 65537 parameters and raises exact accuracy on 24-line reference chains from 15.5% to 99%; a longer-trained version reaches 50 lines in one pass. In Ouro-1.4B, applying the same kind of adapter through recurrent loops extends reach to 60 lines after four loops and at least 160 after eight. Mechanistic tracing suggests the LoRA does not perform token-to-token communication itself; instead, it activates a relay in existing attention and MLP layers, where program lines progressively carry chain identity and ancestor names through a short middle-layer window. Parent-line attention is necessary, while removing it after the relay has formed has little effect. The placement is sharply constrained: moving the Qwen3-8B intervention from layer 20 to layer 21 reduces reach from 20.5 to 5.2 lines. On the multi-hop MuSiQue benchmark, early task-specific LoRAs improve exact-match performance by 11.4 points on Qwen3-8B, 9.4 on OLMo-3-7B, and 17.9 on Llama-3.1-8B, showing that the intervention exposes computation already latent in frozen models rather than simply adding capacity.