You don't need long context
AuthorsPeter Belcak, Saurav Muralidharan, Jindong Jiang, Yonggan Fu, Wonmin Byeon, Hanrong Ye, Ryan Angilly, Ali Taghibakhshi, Shih-Yang Liu, Siru Ouyang, Greg Heinrich, Pavlo Molchanov
The paper finds that language models can often work effectively with short, frequently refreshed contexts, potentially making long-context processing less necessary and costly.
Key results
Fold improvement when reducing the configured context from 256K to 16K for Qwen 3 Coder Next.
At most 5 of 22 tested compaction configurations lost more than 2 percentage points.
Percentage-point gain with selective input reading and a 16K context limit.
Percentage-point gain with selective input reading and a 16K context limit.
Median peak live context was 7–9K tokens across the tested models and benchmarks.
What the paper found
This paper tests whether long-context language models need to keep entire histories or documents in their active context. In experiments with Claude Opus 4.5 and OpenAI GPT 5.6 Sol, end-of-conversation compaction broadly preserved accuracy on long-horizon tasks; at most 5 of 22 tested configurations lost more than 2 percentage points. For long-input tasks, agents selectively searched files with Bash tools rather than ingesting whole documents, often matching or improving on full-input results. On BABILong, this approach raised accuracy by 24.9 percentage points for Claude Opus 4.5 and 23.3 for GPT 5.6 Sol. Even single-action compaction—one tool action per context, followed by a compact semantic-state update—kept median peak contexts to 7–9K tokens. The authors propose Semantic State Space Models, which carry an explicit natural-language state between actions instead of a growing conversation history. Serving measurements for GLM 5.2, Kimi K2.6, and Qwen 3 Coder Next on NVIDIA H100s found that reducing the configured context from 256K to 16K improved decode throughput by up to 7-fold. The results support short-context workflows, though they do not prove every task can avoid long context, and some open-weight models were less robust to aggressive compaction.
Original abstract
Later tokens are costlier to generate than earlier tokens. On frontier open weight models, the disparity in cost may rise to as much as 7x. Through frequent compaction, we show across a broad range of long-horizon and long-input tasks that frontier language models sacrifice little to no task accuracy when maintaining short contexts. We further demonstrate that LLM performance does not collapse on long-horizon software engineering tasks even when taking at most one action per context, and based on that we outline the concept of semantic state space models (SSSMs). Our findings question the need for long-context language models.
Read the original paperMore in Efficient AI
Browse all 79 papers →Latent Feedback Is a Compute and Data Amplifier
Benhao Huang, Chaitanya Dwivedi, Xuezhe Ma, Alexander Moreno, Hector Liu
Latent feedback may let language models get more value from each training token and FLOP than simply repeating the data.
MACaQ: Fast, Model Agnostic Calibrated Quantization for Low-Precision Weights and Activations
Lorenz K. Muller, Philippe Bich, Chiara Boretti, Hyun-Min Chang, Jiawei Zhuang, Lukas Cavigelli
MACaQ aims to make large neural networks cheaper to run by quantizing their weights and activations quickly, with little added inference overhead.
Mobile AI Factory: Building Specialized Language Models for Your Mobile Device
Evan Li, Krishna Deshpande, Wen Cheng, Shyamnath Gollakota
Mobile AI Factory automatically builds and tunes a small language model for a specific task and phone, aiming to deliver strong performance with less compute.