NTH

You don't need long context

AuthorsPeter Belcak, Saurav Muralidharan, Jindong Jiang, Yonggan Fu, Wonmin Byeon, Hanrong Ye, Ryan Angilly, Ali Taghibakhshi, Shih-Yang Liu, Siru Ouyang, Greg Heinrich, Pavlo Molchanov

November 11, 2026 2 min read
Watch on YouTube
The one-line take

The paper finds that language models can often work effectively with short, frequently refreshed contexts, potentially making long-context processing less necessary and costly.

Key results

7
Maximum decode-throughput gain

Fold improvement when reducing the configured context from 256K to 16K for Qwen 3 Coder Next.

5
Long-horizon configurations losing over 2 percentage points

At most 5 of 22 tested compaction configurations lost more than 2 percentage points.

24.9
BABILong accuracy gain, Claude Opus 4.5

Percentage-point gain with selective input reading and a 16K context limit.

23.3
BABILong accuracy gain, GPT 5.6 Sol

Percentage-point gain with selective input reading and a 16K context limit.

9K
Maximum median peak context under single-action compaction

Median peak live context was 7–9K tokens across the tested models and benchmarks.

What the paper found

This paper tests whether long-context language models need to keep entire histories or documents in their active context. In experiments with Claude Opus 4.5 and OpenAI GPT 5.6 Sol, end-of-conversation compaction broadly preserved accuracy on long-horizon tasks; at most 5 of 22 tested configurations lost more than 2 percentage points. For long-input tasks, agents selectively searched files with Bash tools rather than ingesting whole documents, often matching or improving on full-input results. On BABILong, this approach raised accuracy by 24.9 percentage points for Claude Opus 4.5 and 23.3 for GPT 5.6 Sol. Even single-action compaction—one tool action per context, followed by a compact semantic-state update—kept median peak contexts to 7–9K tokens. The authors propose Semantic State Space Models, which carry an explicit natural-language state between actions instead of a growing conversation history. Serving measurements for GLM 5.2, Kimi K2.6, and Qwen 3 Coder Next on NVIDIA H100s found that reducing the configured context from 256K to 16K improved decode throughput by up to 7-fold. The results support short-context workflows, though they do not prove every task can avoid long context, and some open-weight models were less robust to aggressive compaction.

Original abstract

Later tokens are costlier to generate than earlier tokens. On frontier open weight models, the disparity in cost may rise to as much as 7x. Through frequent compaction, we show across a broad range of long-horizon and long-input tasks that frontier language models sacrifice little to no task accuracy when maintaining short contexts. We further demonstrate that LLM performance does not collapse on long-horizon software engineering tasks even when taking at most one action per context, and based on that we outline the concept of semantic state space models (SSSMs). Our findings question the need for long-context language models.

Read the original paper

More in Efficient AI

Browse all 79 papers →
01Efficiency

Latent Feedback Is a Compute and Data Amplifier

Benhao Huang, Chaitanya Dwivedi, Xuezhe Ma, Alexander Moreno, Hector Liu

Latent feedback may let language models get more value from each training token and FLOP than simply repeating the data.

Read analysis