Cortex: Consolidating Context into Parametric Memory
AuthorsJintian Zhang, Feng He, Xinnian Liang, Ziwen Xu, Haoming Xu, Haojing Huang, Shuangzhi Wu, Ningyu Zhang
AffiliationsZhejiang University · ByteDance Seed · Tsinghua University
Cortex aims to help language models retain useful information from temporary context without forgetting what they already know.
Key results
Updates applied to Qwen2.5-14B-Instruct.
Context-induced gain recovered after the sequential edits.
Original non-target performance retained after the sequential edits.
Trainable Structural parameters added for each factual update.
What the paper found
Cortex, introduced by ByteDance researchers, tackles a key limitation of in-context learning: useful behavior disappears when its supporting context is removed. It turns that behavior into parametric memory by generating prospective queries, then training a context-free student to match a context-conditioned teacher using output-distribution KL and hidden-state MSE. Updates are restricted to feed-forward networks: Synaptic adds low-rank weight updates, while Structural adds mergeable nonlinear channels; for mixture-of-experts models, a shared branch gives every token access to the new parameters. Historical query replay helps protect earlier learning without retaining old source contexts. The five-family testbed covers agent experience, novel APIs, persona steering, knowledge suppression, and factual editing, including ALFWorld, LiveCodeBench, and RWKU. Across models including Qwen and OpenAI’s GPT-OSS, Cortex generally reaches the consolidation–retention Pareto frontier. On Qwen2.5-14B-Instruct, after 50 sequential factual edits, Structural recovers 88.9% of the context-induced gain and retains 99.8% of original non-target performance, using 0.74M new parameters per edit.
Original abstract
Context can improve large language model performance, but these gains may be lost after context removal. Inspired by biological systems consolidation, we formulate Memory Consolidation (MC) as internalizing these gains while preserving non-target capabilities, connecting context-specified editing, skill learning, and behavioral steering under a common objective. We introduce a five-task-family testbed measuring consolidation and retention under single, batch, and sequential updates. We propose Cortex, which generates prospective queries to anticipate uses of context and distills context-conditioned outputs into mergeable low-rank or structural feed-forward network updates. Across 4B--120B dense and mixture-of-experts backbones, including both text-only and multimodal models, Cortex lies on the empirical consolidation--retention Pareto frontier in most evaluated settings. After 50 sequential factual edits on a 14B model, Cortex-Structural recovers 88.9% of the context-induced gain while retaining 99.8% of original non-target performance.
Read the original paperMore in Continual Learning
Browse all 27 papers →MemT-1.0: Towards Memory-Native Evolution
Zhiyu Li, Zeyu Zhang, Linyun Liu, Yiwei Bu, Chenyang Xi, Xichong Zhang, Yi Yu, Yu Yu, Zhichen Liu, Yuxiao Yi, Chunyu Li, Haoran Liu, Zhenkai Ding, Le Zhou, Juncheng Zhang, Boyu Chen, Zehao Lin, Guorui Li, Renyu Fu, Ji Li, Youyang Xia, Jiajun Shen, Harvey Li, Jingsong Zhou, Helan Hu, Tong Shen, Tianxiao Xu, Chenye Han, Weijie Li, Zhongquan Zhou, Hongming Xu, Yuekeng Li, Erenkai Zhu, Peng Yu, Wenhao Li, Jingyi Kang, Xixuan Hao, Xuzhou Zhu, Can Chen, Jiawei Liu, Chen Jiang, Tao Jiang, Fangwei Xiang, Zepeng Yao, Kai Chen, Guang Yang, Mingfei Lu, Chunhui Li, Zhengxuan Jiang, Bo Tang, Shiyu Zhan, Yihang Sun, Jie Chu, Wenjun Xiong, Zhenxiang Hua, Chen Lin, Detong Su, Dongyu Deng, Qiuping Ding, Guofeng Ding, Si Chen, Guohua Wen, Xu Chen, Zhi-Qin John Xu, Ying-Cong Chen, Ningyu Zhang, Haofen Wang, Muning Wen, Haotian Chen, Wentao Zhang, Xuanhe Zhou, Rita Zhang, Junchi Yan, Feiyu Xiong
MemT-1.0 builds persistent memory into language models to help them retain and use information more efficiently over long histories.
Selection Is Not Learning: Choosing Model Futures
Haoyi Yang, Kshitij Mishra, Nils Lukas, Salem Lahlou
A framework for self-evolving language models that checks whether an update actually helps before inheriting it, while retaining useful experience from other updates.
ASCENT: Online Test-Time Training of Long-Horizon Agents via Self-Distillation of Verified Experience
Haodong Lu, Dong Gong
ASCENT lets deployed LLM agents learn from verified successes on the fly by converting hindsight about their own trajectories into lasting weight updates.