NTH

Muir: Linear Attention with Sensitivity-Allocated Gating and Matched-Filter Readout

AuthorsTushaar Gangavarapu, Gil Tabak, Clemens JS Schaefer

AffiliationsUniversity of Texas at Austin · \thepa

November 15, 2026 2 min read
Watch on YouTube
The one-line take

Muir redesigns linear attention’s gates and readout to improve language-model quality and long-context retrieval while preserving efficient training and inference.

Key results

20.4
Validation loss improvement over GDN-2

Millinats lower at 1.34B parameters after 100B FineWeb-Edu training tokens.

30.4%
Feed-forward network width increase

Increase over GDN-2 at the same parameter budget.

2.74
Training speedup over Transformer

Times faster at 32K sequence length on Google TPU7x.

10M
Long-stream extrapolation length

Dynamic-readout Muir extrapolates from 4K training sequences to streams of this length.

99.4%
Passkey retrieval accuracy

Static-readout Muir after cooldown at 1M tokens.

What the paper found

Muir redesigns linear attention by exploiting an asymmetry in gated delta-rule memory: writes affect every value channel, while erasure mostly affects the sparse subset of key channels that are active. It uses sensitivity-allocated low-rank gates, a matched-filter readout that reweights state rows according to their signal and interference, and a wider feed-forward network; a boundary-pivoted chunkwise algorithm stabilizes training at large chunk sizes. Trained on FineWeb-Edu with the Llama-2 tokenizer, Muir at 1.34B parameters lowers validation loss by 20.4 millinats versus GDN-2 and widens the feed-forward network by 30.4 percent at the same parameter budget. On Google TPU7x, it trains 2.74 times faster than a Transformer at 32K sequence length. Dynamic readout supports extrapolation from 4K training sequences to 10M-token streams, while the static-readout variant reaches 99.4 percent passkey retrieval at 1M tokens after cooldown. Together, these results show how allocating gate capacity by sensitivity and adapting readout can improve both memory use and long-context retrieval without sacrificing throughput.

Original abstract

Linear attention compresses context into a fixed-size key-value state for linear-time training and constant-memory inference. Gated delta-rule models decay this state and erase its prediction for an incoming key before writing a new value; Gated DeltaNet-2 (GDN-2) generalizes this with channel-wise decay and decoupled, full-rank erase and write gates. However, we show that state updates are far less sensitive to erasure than to writes: key activation and unit normalization concentrate key energy onto a sparse subset of channels, whereas value writes scale all channels, so the erase gate can be parameterized at lower rank. Furthermore, slow-decaying state rows accumulate much larger norms than fast rows, dominating unweighted readout and forcing stronger write-time erasure to suppress interference. Guided by this analysis, we introduce Muir, which fuses all gates into a sensitivity-allocated low-rank projection, re-weights state rows with a matched-filter readout gate, and reinvests the saved parameters in a wider feed-forward network. Because standard chunkwise algorithms become numerically unstable at the large chunk sizes needed to saturate wide matrix units, we design a parallel update for stable, efficient training. Across 166M-1.34B parameters, Muir lowers perplexity by 1.8-2.0% over GDN-2 and 2.5-3.6% over Transformer, raises commonsense accuracy by up to 1.9 points, and improves extractive recall over GDN-2. At 1.34B, Muir outperforms both baselines on multi-hop and multi-needle recall, extrapolates 2,560x from its 4K training length to 10M-token streams without perplexity drift, and maintains near-100% passkey retrieval to 1M tokens with static readout. Muir achieves these gains while matching GDN-2 throughput, reaching its 100B-token validation loss 1.2x faster, and training 2.7x faster than Transformer at 32K.

Read the original paper

More in Attention Mechanisms

Browse all 27 papers →
03Attention

HyScale: Scaling Long-Context Training for Learned Sparse Attention

Yangshen Deng, Xinjing Huang, Yulong Wang, Yunzhuo Liu, Jun Hu, Shuyong Tan, Chun-Chih Tseng, Mingji Han, Ray Ying, Key Zhang, Luo Mai

HyScale makes training long-context sparse attention faster by avoiding large selection tables and reusing selection results during recomputation.

Read analysis