Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention
AuthorsAles Kubicek, Andrea Caciolai, Liangzu Peng, Sukrit Sumant, Benjamin Hoover, Fraser Greenlee, Ruairidh M. Battleday, Radim Urban, Torsten Hoefler, James C. R. Whittington
Latent Sparse Attention organizes keys into learned clusters so models can attend over million-token contexts more efficiently.
Key results
D-LSA evaluation on GLM-5.3-F LASH and QWEN-3.8-27B.
Times faster per attention layer than GLM-5.3-F LASH’s original sparse attention.
Times faster per attention layer than GLM-5.3-F LASH’s original sparse attention.
Times faster per attention layer than full attention.
Times faster per attention layer than full attention.
Times faster per attention layer at 128k tokens, matching full-attention accuracy.
What the paper found
As Google’s Gemini, DeepSeek, and Qwen push toward million-token contexts, the cost of finding useful keys can overwhelm the savings from sparse attention. Latent Sparse Attention, or LSA, tackles that bottleneck by pooling keys, clustering them by content in a learned latent space, and having each query score cluster centroids rather than every key. Its tree-search variant reduces prefill indexing to O(T log T). Distilled LSA, or D-LSA, adds small learned projections to pretrained models; Native LSA, or N-LSA, learns directly from next-token prediction without extra parameters or a teacher. At 10M tokens, D-LSA on GLM-5.3-F LASH makes one attention layer’s prefill 22.7× and decode 4.6× faster than the model’s original sparse attention; on QWEN-3.8-27B, it is 72.4× faster in prefill and 42.5× in decode than full attention, while staying close to baseline accuracy. N-LSA matches full-attention accuracy and speeds up one layer’s training by 1.6× at 128k tokens. The work also integrates LSA into SGLang and reports GPU implementations for NVIDIA H200 and B300 accelerators.
Original abstract
Agentic workflows are increasingly shifting towards long-horizon tasks, such as open-ended scientific work, where models benefit from long context windows. However, at ultra-long context (above 1M tokens), full attention is prohibitively expensive. A promising approach is sparse attention, in which a dedicated indexer selects a subset of keys for the attention. Yet current methods of indexing either still have quadratic cost in context size or lose accuracy at ultra-long context scale. We address this with Latent Sparse Attention (LSA), an efficient attention method that maintains accuracy at ultra-long context. In LSA, keys are hierarchically clustered in a learned latent space, and queries score the hierarchy of clusters instead of scoring every key, making prefill, decode and training faster at long context. We propose two variants, Distilled-LSA (D-LSA), which can be applied to pre-trained models via distillation, and Native-LSA (N-LSA), which is trained natively on the next-token loss alone. On GLM-5.3-FLASH (QWEN-3.8-27B) at 10M tokens, D-LSA speeds up one attention layer by 22.7x (72.4x) in prefill and 4.6x (42.5x) in decode over its original sparse attention (full attention), while keeping similar accuracy across a variety of tasks. N-LSA matches the accuracy of full attention with a training speedup of 1.6x per layer at 128k tokens. LSA offers practical and efficient attention for models to benefit from ultra-long context.
Read the original paperMore in Attention Mechanisms
Browse all 24 papers →HyScale: Scaling Long-Context Training for Learned Sparse Attention
Yangshen Deng, Xinjing Huang, Yulong Wang, Yunzhuo Liu, Jun Hu, Shuyong Tan, Chun-Chih Tseng, Mingji Han, Ray Ying, Key Zhang, Luo Mai
HyScale makes training long-context sparse attention faster by avoiding large selection tables and reusing selection results during recomputation.
Subquadratic Attention with a Hierarchical Tree-Structured Indexer
Leon Lufkin, Cutter Dawes, Alexander Speicher, Kamesh Krishnamurthy, Tomás Figliolia, Beren Millidge
ZHA uses a hierarchical tree to find relevant keys efficiently, aiming to make very long-context attention practical.
Can Computation from Earlier Problems Help LLMs Solve New Ones?
Jipei He, Wenhui Tan, Xiaoyi Yu, Enver Sangineto, Fiorenzo Parascandolo, Rita Cucchiara, Ruihua Song
STAIR helps language models reuse useful computation from earlier questions, improving performance on later problems with only a tiny trainable module.