MemT-1.0: Towards Memory-Native Evolution
AuthorsZhiyu Li, Zeyu Zhang, Linyun Liu, Yiwei Bu, Chenyang Xi, Xichong Zhang, Yi Yu, Yu Yu, Zhichen Liu, Yuxiao Yi, Chunyu Li, Haoran Liu, Zhenkai Ding, Le Zhou, Juncheng Zhang, Boyu Chen, Zehao Lin, Guorui Li, Renyu Fu, Ji Li, Youyang Xia, Jiajun Shen, Harvey Li, Jingsong Zhou, Helan Hu, Tong Shen, Tianxiao Xu, Chenye Han, Weijie Li, Zhongquan Zhou, Hongming Xu, Yuekeng Li, Erenkai Zhu, Peng Yu, Wenhao Li, Jingyi Kang, Xixuan Hao, Xuzhou Zhu, Can Chen, Jiawei Liu, Chen Jiang, Tao Jiang, Fangwei Xiang, Zepeng Yao, Kai Chen, Guang Yang, Mingfei Lu, Chunhui Li, Zhengxuan Jiang, Bo Tang, Shiyu Zhan, Yihang Sun, Jie Chu, Wenjun Xiong, Zhenxiang Hua, Chen Lin, Detong Su, Dongyu Deng, Qiuping Ding, Guofeng Ding, Si Chen, Guohua Wen, Xu Chen, Zhi-Qin John Xu, Ying-Cong Chen, Ningyu Zhang, Haofen Wang, Muning Wen, Haotian Chen, Wentao Zhang, Xuanhe Zhou, Rita Zhang, Junchi Yan, Feiyu Xiong
MemT-1.0 builds persistent memory into language models to help them retain and use information more efficiently over long histories.
Key results
Tokens used across continued pre-training and post-training.
Average score across 11 memory benchmarks.
Score-point advantage for Flash over the strongest parametric-memory baseline.
Nano’s end-to-end inference speedup versus full-context inference.
Fold reduction in persistent storage versus the full-context KV cache.
Nano’s average success with harness-triggered experience storage.
What the paper found
MemT-1.0 makes persistent memory part of a language model’s computation rather than relying on a growing prompt or an external retrieval system. Its architecture uses stable token-level updates based on Gated DeltaNet-2 and Mixture-of-Storage, which densely routes information across multiple memory experts. A two-stage training pipeline—continued pre-training followed by post-training with full-context teacher distillation—uses 34B tokens to teach the model what to retain and how to retrieve it. The released Nano and Flash models build on Qwen3.6-35B-A3B and DeepSeek-V4-Flash, respectively, with training infrastructure using NVIDIA’s Megatron-Core and Bridge. Across memory benchmarks, Flash scores 50.6 on average, 12.5 points ahead of the strongest parametric-memory baseline; explicit textual-memory systems can still outperform it, underscoring the challenge of preserving detailed evidence in a fixed-size state. At 128K tokens of history, Nano delivers 1.86× faster end-to-end inference and 66× less persistent storage than a full-context KV cache. The paper also tests procedural memory in OpenAI’s SWE-bench Verified: when experience is stored after a detected failure, Nano reaches 69.73% average success, versus 66.07% without experience. The results point toward agents that can reuse past experience efficiently, while leaving memory fidelity and autonomous experience acquisition as open challenges.
Original abstract
Memory-native evolution requires making memory an intrinsic capability of foundation models, enabling them to continuously accumulate and utilize information. To this end, we introduce MemT-1.0, a new architecture of memory foundation models that internalize memory as part of model computation. Building on Metis, MemT-1.0 employs Gated DeltaNet-2 with a stable token-level memory update mechanism within each turn. We further introduce Mixture-of-Storage to scale native memory capacity. Following a deliberate backbone preparation, we develop a two-stage training pipeline consisting of continued pre-training and post-training, supported by 34B tokens of training data and efficient training and inference infrastructure. We release a model family comprising MemT-1.0-Nano and MemT-1.0-Flash, and conduct a comprehensive evaluation of memory and general capabilities. The results show that they outperform the parametric memory baselines on average and remain competitive with retrieval-augmented methods, while largely preserving general capabilities. At a history length of 128K tokens, MemT-1.0-Nano achieves approximately $1.86\times$ end-to-end inference speedup and a $66\times$ reduction in persistent storage relative to full context KV cache. By internalizing retrieved procedural experience, MemT-1.0 improves task success on agentic tasks, providing a practical step toward memory-native evolution.
Read the original paperMore in Continual Learning
Browse all 27 papers →Cortex: Consolidating Context into Parametric Memory
Jintian Zhang, Feng He, Xinnian Liang, Ziwen Xu, Haoming Xu, Haojing Huang, Shuangzhi Wu, Ningyu Zhang
Cortex aims to help language models retain useful information from temporary context without forgetting what they already know.
Selection Is Not Learning: Choosing Model Futures
Haoyi Yang, Kshitij Mishra, Nils Lukas, Salem Lahlou
A framework for self-evolving language models that checks whether an update actually helps before inheriting it, while retaining useful experience from other updates.
ASCENT: Online Test-Time Training of Long-Horizon Agents via Self-Distillation of Verified Experience
Haodong Lu, Dong Gong
ASCENT lets deployed LLM agents learn from verified successes on the fly by converting hindsight about their own trajectories into lasting weight updates.