Parallax: Parameterized Local Linear Attention for Language Modeling
AuthorsYifei Zuo, Dhruv Pai, Zhichen Zeng, Alec Dewulf, Shuming Hu, Zhaoran Wang
Resources
Parallax is a new attention design for language models that makes local linear attention practical at scale and claims better training quality and faster decoding than FlashAttention-style baselines.
Key results
Transformer baseline average downstream accuracy at 0.6B scale.
Parallax with Muon at 0.6B scale reaches this average downstream accuracy, improving over the Transformer baseline.
Transformer baseline average downstream accuracy at 1.7B scale.
Parallax with Muon at 1.7B scale reaches this average downstream accuracy, improving over the Transformer baseline.
What the paper found
Parallax, from Northwestern University, Tilde Research, and the University of Washington, rethinks attention for language modeling by parameterizing Local Linear Attention instead of solving its per-token ridge system exactly. The paper starts from the test-time regression view of attention and shows that Local Linear Attention improves the bias-variance tradeoff over softmax attention, but is too expensive and numerically fragile for large-scale pretraining. Parallax removes the conjugate-gradient solve and learns an extra query-like projector, R, that probes KV covariance, while also introducing a hardware-aware streaming kernel that reuses the same KV tiles as FlashAttention and doubles arithmetic intensity in the large-context regime. On NVIDIA H200, the prototype decode kernel matches or beats FlashAttention 2 and 3 across batch sizes 1 to 2,048 and contexts up to 32,768. In pretraining at 0.6B and 1.7B parameters on Ultra-FineWeb with a Qwen-3 backbone, Parallax consistently lowers perplexity and improves downstream accuracy on LAMBADA, WikiText, BoolQ, HellaSwag, PIQA, ARC-Easy, ARC-Challenge, WinoGrande, OpenBookQA, and SciQ; the 0.6B Muon run reaches 55.99 average accuracy versus 54.54 for the Transformer baseline, and the 1.7B Muon run reaches 62.45 versus 61.43. A key finding is optimizer-architecture codesign: the gains are strong under Muon but largely vanish under AdamW, because Muon increases the norm and alignment of the correction branch, allowing the covariance term to activate instead of collapsing toward softmax attention.
Original abstract
Large Language Models (LLMs) have become the central paradigm in artificial intelligence, yet the core computational primitive of attention has remained structurally unchanged. Local Linear Attention (LLA) is an attention mechanism derived from nonparametric statistics in the test-time regression framework. In contrast to prior research on efficient attention variants, LLA upgrades the local constant estimate in softmax attention to a local linear estimate, yielding provably superior bias-variance tradeoffs for associative memory. However, LLA has not been scaled in LLM pretraining due to computational and numerical stability concerns. We introduce Parallax, a parameterized Local Linear Attention that is scalable for LLMs. Parallax eliminates the numerical solver in LLA and learns an extra query-like projector that probes the KV covariance. We place Parallax within a family of attention mechanisms connected by the bandwidth, the probe construction and the affine structure. We propose a hardware-aware algorithm that increases the arithmetic intensity over FlashAttention, shifting attention into a more compute bound regime. Our prototype decode kernel matches or outperforms FlashAttention 2/3 across diverse batch sizes and context lengths. We pretrain Parallax at 0.6B and 1.7B scales and find consistent perplexity improvements throughout pretraining with gains that transfer to downstream benchmarks. The advantage persists under both parameter-matched and compute-matched controls, demonstrating a Pareto improvement. We perform careful pretraining ablations and identify a novel phenomenon whereby Muon unlocks the capacity of Parallax. To our knowledge, this is the first empirical demonstration of strong architecture-optimizer codesign for attention mechanisms in the architecture research literature.
Read the original paperMore in Transformers
Browse all 42 papers →Pretraining Latent Information Feedback Transformers with Teacher Supervision
Dor Tirosh, Ido Amos, Mor Geva
LIFT teaches Transformers to pass rich hidden-state information across steps, potentially making language models more efficient and capable than standard feed-forward designs.
The Geometry of Inference in Transformer Residual Streams
Timur Mudarisov, Mikhail Burtsev, Radu State
This paper shows how Transformer hidden states gradually geometrically converge toward the correct prediction while eliminating competing possible outcomes.
Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It
Zehao Jin, Ruixuan Deng, Junran Wang
A small LoRA update appears to make transformers carry information through many more layers, dramatically extending their ability to follow long chains without retraining the full model.