Competing Learning in Hybrid Architectures
AuthorsXingyu Shen, Yingfa Chen, Ziqing Qiao, Chaojun Xiao, Xu Han, Zhiyuan Liu
AffiliationsDepartment of Science and Technology, Tsinghua University, Beijing, China · xingyu-c21@mails.tsinghua.edu.cn, chenyingfa1999@gmail.com
A staged training approach helps hybrid language models balance recurrence and attention, improving language modeling and retrieval.
Key results
Zero-shot macro-average recall score; vanilla reached 36.50%.
Average across 57 task–length combinations; vanilla reached 32.75%.
Selected-token perplexity; vanilla reached 7.69.
Vanilla training loss was 2.8744.
What the paper found
Hybrid language models such as Kimi K3, GLM-5.3-Flash, Qwen3.8-Flash-Next, and NVIDIA’s Nemotron-3-Ultra combine efficient recurrent layers with softmax attention, but those components can compete to learn the same retrieval behavior. The paper shows that Kimi Delta Attention, or KDA, often learns retrieval first because it favors short-range patterns; by reducing retrieval error early, recurrent layers weaken the learning signal for attention. Its proposed Better Layer Division method freezes recurrent blocks during an initial 9.09% of pretraining, giving attention a head start to specialize in long-range retrieval. Tested on DCLM with Qwen3 tokenization, context lengths of 16,384, and models from 180M to 1B parameters, BLD improved retrieval and long-context benchmarks, with similar benefits when recurrent layers were replaced by Gated DeltaNet or Gated Linear Attention. On the 1B model, BLD reached 38.14% Recall versus 36.50% for vanilla training, and 36.04% on RULER versus 32.75%; LongPPL fell to 6.50 from 7.69, while training loss decreased to 2.8680 from 2.8744. The results suggest that training order can help hybrid architectures realize a more effective division of labor without changing their design.
Original abstract
Many modern language models have adopted hybrid architectures that incorporate both linear-time recurrence and softmax attention. Modern recurrence layers provide a highly expressive mixing mechanism while ensuring high throughput by compressing contextual information into a fixed-size state, but underperform softmax attention in contextual information retrieval. In this paper, we reveal a Competing Learning phenomenon where different kinds of mixers compete to learn the same function, and when any layer learns a function, it suppresses the corresponding learning signal for other layers. Layer attribution analysis reveals that under typical data distributions, modern recurrence learns retrieval functions faster than softmax attention, causing attention to fail to learn retrieval in most cases. Stopping weight updates in recurrence in the middle of training helps attention learn retrieval. Based on these results, we propose a staged training procedure for pre-training hybrid architectures on natural language. Empirical results on models up to 1B parameters show that our method results in substantial improvement in language modeling loss, common-sense reasoning, as well as a wide range of retrieval-related tasks across different context lengths.
Read the original paperMore in Large Language Models
Browse all 90 papers →Belief-Trajectory Energy: Measuring the Path to a Prediction
Jiahao Ying, Wei Tang, Boxian Ai, Yaoning Wang, Haotian Chen, Wenhe Sun, Caijun Xu, Haozhan Cai, Changyi Xiao, Yixin Cao
Belief-Trajectory Energy tracks how an LLM’s predictions change across layers to reveal information that its final answer alone can miss.
On the Tip of the Tongue: First-Token Cueing as a Behavioral Probe of Recall Failures in LLMs
Jaewon Sok, Jewon Yeom, Seonghyeon Park, Jeongjae Park, Hwiyeong Lee, Taesup Kim
A single correct opening token can help an LLM recover factual answers it seemed unable to recall.
Copy-Paste Language Models
Nathan Godey, Yoav Artzi
Copy-Paste Language Models add a low-overhead copying mechanism that improves language-model training and performance, especially in long-context and retrieval settings.