Memory for Large Language Models

“Memory for Large Language Models,” by Sining Zhoubian, Dan Zhang, Evgeny Kharlamov of Bosch AI, and Jie Tang, proposes a mechanism-centric survey that treats memory as a first-class architectural dimension rather than an incidental result of scaling. Its taxonomy organizes LLM memory along three orthogonal axes: representation, distinguishing implicit computation-coupled states from explicit addressable storage; update dynamics, separating offline training updates from online inference-time adaptation; and persistence, distinguishing short-term context memory from long-term storage. The survey unifies Transformer KV caches, sparse attention, Mamba and other recurrent state-space models, test-time training systems such as Titans, lookup mechanisms such as Engram, and conditional parameter routing in MoE models including Mixtral and DeepSeek-MoE. It highlights a fundamental trade-off: attention provides high-fidelity token recall but has quadratic sequence-cost scaling, while recurrent models offer linear-time processing and bounded storage by compressing history into latent states, at the cost of information loss and interference. Explicit memory improves persistence and controllability through writable parameters, memory slots, or independent lookup stores, but introduces drift, stability, indexing, and lifecycle-management problems. The survey also discusses DeepSeek’s latent KV compression, Apple’s RATTENTION and CommVQ, hybrid systems such as Jamba and Kimi Linear, and evaluation with RULER, LongBench, ∞Bench, and SCROLLS. Its central conclusion is that future LLMs will require adaptive hybrid memory, multi-timescale updates, hardware–algorithm co-design, and evaluation across capacity, recall fidelity, persistence, robustness, and efficiency.