The Surprising Effectiveness of Shared Memory in Looped Transformers

Looped Transformers reuse the same layers for multiple passes, but normally each pass keeps its own full key-value cache, so memory grows with recursion. This paper introduces the Looped Prediction Transformer, or LPT: the first pass writes a full-context cache, and later passes read it alongside a short local window. Across models from 150M to 1B parameters trained on FineWeb-Edu, this learned memory sharing improves quality: at five recursions, LPT improves accuracy by up to 4.8 percentage points over matched looped Transformers while using up to 4.7× less KV memory. Its hybrid variant, H-LPT, combines sharing with Gated DeltaNet and full-attention layers; at five recursions, it lowers validation perplexity by 1.12–1.82 versus a standard Transformer while using 76–79% less context memory. The authors’ analysis suggests that the first recursion specializes in writing memory, while later recursions mainly use it: as much as 80% of final-recursion attention goes to shared memory. Reading that cache also creates a direct gradient path back to the first recursion, and blocking the path harms quality and can destabilize training. On an NVIDIA H100, H-LPT’s smaller cache lets it fit up to 5.0 times as many sequences per batch as a standard Transformer. Unlike cited cache-reduction work such as DeepSeek-V4.1-Flash, this approach treats memory sharing as a trainable architectural bias—and finds it can improve both quality and memory efficiency.