NTH
Neural Networks

Continuous Memory Machines

AuthorsCiaran Regan, Kai Arulkumaran, Luke Darlow, Stefania Druga, Sebastian Risi, Llion Jones

AffiliationsUniversity of Tsukuba · Sakana AI

October 12, 2026 2 min read
Watch on YouTube

Resources

Read on alphaXiv Read the original paper on arXiv GitHub repository
The one-line take

Continuous Memory Machines give recurrent networks separate short- and long-term memories that can read from and reorganize one another.

Key results

0.0041
Priority-sort test loss

CMM's lowest reported test loss on priority sorting.

72.1%
Length generalization at 50 items

CMM's accuracy on the longest tested input length.

96.3%
Maze action accuracy

CMM accuracy on individual route actions in 39×39 mazes.

61.6%
Maze full-route accuracy

CMM accuracy when the complete predicted route must be correct.

30.63%
Few-shot regression accuracy

CMM accuracy with 20 support examples.

What the paper found

The Continuous Memory Machine, or CMM, addresses a core recurrent-network limitation: one compressed state is often expected to handle both immediate computation and durable storage. Building on the Continuous Thought Machine (CTM), it combines a sliding matrix-valued short-term memory, where neuron-level models process recent activity, with a persistent matrix-valued long-term memory. A Transformer jointly updates both stores, enabling them to read, write, and reorganize information across memory timescales. On priority sorting, CMM reached a test loss of 0.0041, and on length generalization it achieved 72.1% accuracy at 50 input items. In a 39×39 maze-solving task, it scored 96.3% action accuracy and 61.6% full-route accuracy, compared with the CTM’s 88.8% and 37.4%. For few-shot regression with 20 support examples, CMM reached 30.63% accuracy, versus 8.78% for CTM. Attention analysis shows that memory use adapts to the task: the model draws on long-term memory for retention-heavy problems and can suppress memory reads when they are unnecessary. These results suggest that separating short- and long-term memory can improve recurrent reasoning and generalization, though the joint Transformer update increases computational and memory costs.

Original abstract

Recurrent neural networks typically compress information into a single vector-valued recurrent state, forcing short-term computation and long-term retention to share the same representation. Past extensions alleviate this bottleneck by increasing the memory capacity or separating timescales, but lack the combination of rapid neuron-level processing and longer-term retention found in biology. To that end, we introduce the Continuous Memory Machine (CMM), a recurrent architecture with matrix-valued short- and long-term memory states serving distinct functional roles. Building on the Continuous Thought Machine (CTM), the CMM's short-term memory tracks recent neural activity, with uniquely parameterized neuron-level models learning to use these activity patterns for computation. A persistent long-term memory stores information for later use, with a Transformer jointly updating both memory stores, providing an expressive bidirectional read--write mechanism such that each store can reorganize its own contents and both read from and write to the other. Across algorithmic, in-context learning, and recurrent reasoning tasks, the CMM outperforms a broad suite of baselines, exhibiting stronger generalization than prior memory-augmented networks while preserving the CTM's interpretable attention patterns. Code is available at https://github.com/SakanaAI/continuous-memory-machines.

Read the original paper

More in Neural Networks

Browse all 25 papers →
01Neural Network

Inference and learning in sparse autoencoders as natural gradient flow

Hadi Vafaii, Tejas Rao, David Chanin, Thomas Fel, Jacob L. Yates, Bruno Olshausen, David Klindt, Dileep George, Miguel Lázaro-Gredilla

BeFOND uses natural-gradient inference and learning to recover rare and overlapping features in neural representations more effectively and with less data.

Read analysis