Explore how models select and combine information through attention. Follow research on long contexts, efficient computation, and architectural alternatives.
18 papers · Latest edition October 9, 2026
Three of the latest briefs in this collection. Read the evidence and the original papers alongside them.
CoWindow Attention makes long-context transformers faster by letting attention heads collectively cover the past instead of redundantly reading all of it.
HLA makes linear attention more selective by letting each query dynamically choose which compressed chunks of long-context history to access.
MinkowskiPE gives attention a physics-inspired sense of spacetime, improving both molecular dynamics and video prediction with far fewer parameters.
Newest editions first.
Jingze Shi, Zhangyang Peng, Xianduo Li, Yanlin Qi, Xiaotian Lin, Haoxian Chen, Liangdong Wang, Guang Liu, Yuyu Luo
CoWindow Attention makes long-context transformers faster by letting attention heads collectively cover the past instead of redundantly reading all of it.
Zhuokun Chen, Xi Lin, Xiyu Wu, Jiahao He, Jianfei Cai, Bohan Zhuang
HLA makes linear attention more selective by letting each query dynamically choose which compressed chunks of long-context history to access.
Yuhao Li, Louie Hong Yao, Tianyi Shi, Hanqun Cao, Hongxia Hao, Zhen Zhao, Shengchao Liu
MinkowskiPE gives attention a physics-inspired sense of spacetime, improving both molecular dynamics and video prediction with far fewer parameters.
Bohao Tang, Zhen Qin, Yuqi Pan, Zheng Li, Pengfei Liu
PISA makes long-context attention more scalable by hierarchically narrowing relevant key blocks instead of comparing every query with every block.
Jiale Kang
Memory Attention replaces some costly attention computation with reusable token memories, potentially making language models cheaper to run and easier to fit on limited GPU hardware.
Shuyang Xiang
The study finds that real paragraph structure is revealed not merely by where attention concentrates, but by how deeply that compression develops across model layers.
Yizhou Xu, Margarita Sagitova, Lenka Zdeborová, Florent Krzakala
This work explains how different ways of parameterizing attention can determine whether large models learn useful structure or remain stuck, using a detailed high-dimensional theory.
Ngoc Bui, Tinglin Huang, Rex Ying
This work makes linear attention more adaptive by letting its memory track not only what it knows, but also how confident it is.
Yiheng Jiang, Yuanbo Xu, Yongjian Yang
LGA is an efficient attention mechanism that models irregular, long-range time-series dynamics without sequential numerical solvers.
Qucheng Gao, Zuyi Yang, Xiao Chen
This paper argues that self-attention can spontaneously organize token representations into stable clusters once attention becomes sufficiently sharp.
Burc Gokden
This work argues that a power-law, graph-based form of attention can mathematically subsume standard attention and may effectively collapse to it during inference.
Zeqi Zheng, Zizheng Zhu, Yuping Yan, Wenxuan Pan, Zhaofei Yu, Yaochu Jin
This paper makes spiking transformers more spatially aware by combining localized attention with a lightweight pathway that preserves communication across region boundaries.
Tommaso Cerruti, Tim Rieder, George Rowlands, Lingfeng Jin, Imanol Schlag
This paper compares several modern linear-attention designs under one framework and finds that a simple cross-layer value-routing trick can modestly improve performance.
Xin Wang, Hui Shen, Boyuan Zheng, Xueshen Liu, Minkyoung Cho, Zhongwei Wan, Zesen Zhao, Zhuoqing Mao, Shen Yan, Mi Zhang
This paper makes linear attention smarter by dynamically deciding how to merge memory states, aiming to keep long-context models efficient without losing important information.
Zhentao Tan, Wei Chen, Jingyi Shen, Yao Liu, Xu Shen, Yue Wu, Jieping Ye
HydraHead rethinks hybrid attention by assigning full attention only to the most retrieval-critical heads, improving long-context performance while keeping compute efficient.
Aryo Pradipta Gema, Beatrice Alex, Pasquale Minervini
This paper proposes LOCOS, a new way to find attention heads that retrieve meaning rather than just copied words, and shows those heads are crucial for long-context question answering.
Waleed Razzaq, Yun-Bo Zhao
This paper introduces a biologically inspired attention mechanism that turns attention scores into stochastic variables to produce uncertainty-aware predictions across several continuous-time tasks.
Piotr Frydrych
This paper replaces softmax attention with a hysteresis-based memory model that tracks extrema, aiming to give transformers a more efficient way to remember long histories.