Metis: Memory Foundation Model
AuthorsZeyu Zhang, Ziliang Guo, Yihang Sun, Xichong Zhang, Xixuan Hao, Zehao Lin, Yang Zhang, Xiaoyan Zhao, Tong Shen, Bo Tang, Zhi-Qin John Xu, Junchi Yan, Haofen Wang, Xu Chen, Feiyu Xiong, Zhiyu Li, Tat-Seng Chua
Resources
Metis gives foundation models an internal, continually updating memory that can store and recall experience without changing the model’s weights.
Key results
Synthesized memory-training samples spanning remember, update, forget, and reflect operations.
Token volume of the synthesized corpus derived from 27 public benchmarks.
Average no-context score on the MemOps (Gold) memory-operation benchmark.
Average no-context score on the NextMem memory-based QA benchmark.
Metis reduction in LoCoMo (Gold) P95 end-to-end latency versus full-context processing.
Average performance recovered after compressing Metis memory states to rank 64.
What the paper found
Metis, developed by MemTensor with researchers from Renmin University of China, the National University of Singapore, Shanghai Jiao Tong University, and Tongji University, proposes a memory foundation model that internalizes agent memory inside a language model rather than using external RAG storage. Built on Qwen3.5 backbones, Metis adds local memory blocks for dense dynamic states and hyper memory blocks that learn storage procedures through memory attention, adaptive aggregation, and Gated Delta Network updates. During mid-training, the frozen backbone learns memory reconstruction, remember, update, forget, reflect, and robustness behaviors from 357,137 samples and 406.1M tokens synthesized from 27 public benchmarks, while online memory updates require only a forward pass and no gradients. In no-context evaluation, Metis-27B reaches 24.76 on MemOps (Gold), 26.74 on LoCoMo (Gold), and 50.82 on NextMem, outperforming the paper’s parametric-memory and test-time-training baselines, although full-context models remain stronger. Efficiency improves because memory reads and writes can run in parallel and historical text is not replayed: on LoCoMo (Gold), Metis reduces full-context P95 end-to-end latency by 69.3%. Low-rank analysis shows that rank 64 recovers 99.9% of full-state performance, indicating substantial redundancy. The main limitations are compression loss, long-trajectory interference, and degraded general capabilities after irrelevant information accumulates, so Metis is presented as a complementary alternative to external memory rather than a complete replacement.
Original abstract
Recent advances in AI agents have increasingly internalized native capabilities into their underlying foundation models, giving rise to multimodal foundation models and large reasoning models. However, agent memory is still primarily implemented through external modules, leaving the native memory capability largely unexplored. In this paper, we take a first step toward this direction by introducing memory foundation models, which empower foundation models with native memory capabilities. We formalize native memory from two perspectives: a persistent and dynamically evolving memory state within the backbone, and native memory procedures that autonomously store and utilize information through model computation. We show that native memory offers advantages in architecture, end-to-end optimization, and efficiency. Based on this formulation, we propose Metis, the first prototype of memory foundation models. Metis introduces a new architecture that equips a foundation model with a native memory state, allowing historical information to be compressed into the model and accessed through memory attention. We construct large-scale memory-specific training data and introduce multiple optimization objectives to acquire these native memory procedures through mid-training. The online memory maintenance of Metis is gradient-free, and the memory update requires only a forward pass. At inference time, all learned model weights remain frozen, while the native memory states are autonomously transformed through standard forward computation. Through extensive experiments, we show that Metis exhibits native memory capabilities and further provide a detailed analysis of its strengths, limitations, and behaviors. To facilitate future research on memory foundation models, we release our project and model checkpoints.
Read the original paperMore in Continual Learning
Browse all 24 papers →ASCENT: Online Test-Time Training of Long-Horizon Agents via Self-Distillation of Verified Experience
Haodong Lu, Dong Gong
ASCENT lets deployed LLM agents learn from verified successes on the fly by converting hindsight about their own trajectories into lasting weight updates.
From Knowledge Access to Source Learning: Developing Source-Specific Competence
Lucheng Fu, Kejing Xia, Yiyang Wang, Yiqiao Jin, Jinjin He, Xiyuan Yang, Haoxin Liu, Ye Yu, Haibo Jin, Yijia Xiao, Wenke Lee, B. Aditya Prakash, Haohan Wang
SourceLearn helps LLM agents progressively build reusable expertise about trusted information sources instead of repeatedly starting from scratch.
Local Support Learning
Assaf Ben-Kish, Akarsh Kumar, James Glass, Raja Giryes
Local Support Learning helps large language models learn new skills without overwriting what they already know by activating updates only where they are locally needed.