Muir: Linear Attention with Sensitivity-Allocated Gating and Matched-Filter Readout
AuthorsTushaar Gangavarapu, Gil Tabak, Clemens JS Schaefer
AffiliationsUniversity of Texas at Austin · \thepa
Muir redesigns linear attention’s gates and readout to improve language-model quality and long-context retrieval while preserving efficient training and inference.
Key results
Millinats lower at 1.34B parameters after 100B FineWeb-Edu training tokens.
Increase over GDN-2 at the same parameter budget.
Times faster at 32K sequence length on Google TPU7x.
Dynamic-readout Muir extrapolates from 4K training sequences to streams of this length.
Static-readout Muir after cooldown at 1M tokens.
What the paper found
Muir redesigns linear attention by exploiting an asymmetry in gated delta-rule memory: writes affect every value channel, while erasure mostly affects the sparse subset of key channels that are active. It uses sensitivity-allocated low-rank gates, a matched-filter readout that reweights state rows according to their signal and interference, and a wider feed-forward network; a boundary-pivoted chunkwise algorithm stabilizes training at large chunk sizes. Trained on FineWeb-Edu with the Llama-2 tokenizer, Muir at 1.34B parameters lowers validation loss by 20.4 millinats versus GDN-2 and widens the feed-forward network by 30.4 percent at the same parameter budget. On Google TPU7x, it trains 2.74 times faster than a Transformer at 32K sequence length. Dynamic readout supports extrapolation from 4K training sequences to 10M-token streams, while the static-readout variant reaches 99.4 percent passkey retrieval at 1M tokens after cooldown. Together, these results show how allocating gate capacity by sensitivity and adapting readout can improve both memory use and long-context retrieval without sacrificing throughput.
Original abstract
Linear attention compresses context into a fixed-size key-value state for linear-time training and constant-memory inference. Gated delta-rule models decay this state and erase its prediction for an incoming key before writing a new value; Gated DeltaNet-2 (GDN-2) generalizes this with channel-wise decay and decoupled, full-rank erase and write gates. However, we show that state updates are far less sensitive to erasure than to writes: key activation and unit normalization concentrate key energy onto a sparse subset of channels, whereas value writes scale all channels, so the erase gate can be parameterized at lower rank. Furthermore, slow-decaying state rows accumulate much larger norms than fast rows, dominating unweighted readout and forcing stronger write-time erasure to suppress interference. Guided by this analysis, we introduce Muir, which fuses all gates into a sensitivity-allocated low-rank projection, re-weights state rows with a matched-filter readout gate, and reinvests the saved parameters in a wider feed-forward network. Because standard chunkwise algorithms become numerically unstable at the large chunk sizes needed to saturate wide matrix units, we design a parallel update for stable, efficient training. Across 166M-1.34B parameters, Muir lowers perplexity by 1.8-2.0% over GDN-2 and 2.5-3.6% over Transformer, raises commonsense accuracy by up to 1.9 points, and improves extractive recall over GDN-2. At 1.34B, Muir outperforms both baselines on multi-hop and multi-needle recall, extrapolates 2,560x from its 4K training length to 10M-token streams without perplexity drift, and maintains near-100% passkey retrieval to 1M tokens with static readout. Muir achieves these gains while matching GDN-2 throughput, reaching its 100B-token validation loss 1.2x faster, and training 2.7x faster than Transformer at 32K.
Read the original paperMore in Attention Mechanisms
Browse all 27 papers →Hard Sample-Conditioned Decorrelated Attention for Robust Face Recognition
Tahar Chettaoui, Guray Ozgur, Eduarda Caldeira, Naser Damer, Fadi Boutros
ViT-HarDis encourages attention heads to specialize more on difficult face samples, improving recognition without extra inference cost.
Training Vision Transformers to Focus: Emergence of Spatial Attention Head Specialization
Yuji Nozawa, Yu-Chieh Lin, Youyang Ng
The method encourages Vision Transformers to develop distinct, spatially focused attention heads without image annotations.
HyScale: Scaling Long-Context Training for Learned Sparse Attention
Yangshen Deng, Xinjing Huang, Yulong Wang, Yunzhuo Liu, Jun Hu, Shuyong Tan, Chun-Chih Tseng, Mingji Han, Ray Ying, Key Zhang, Luo Mai
HyScale makes training long-context sparse attention faster by avoiding large selection tables and reusing selection results during recomputation.