ResKV: Reconstructing Omitted Attention Contributions for Fixed-Budget KV Cache Compression

ResKV, from Yuhang Zhan, Lisi Chen, and Shuo Shang at the University of Electronic Science and Technology of China, addresses a weakness in KV-cache compression: hard eviction discards both the numerator and denominator contributions of omitted tokens, while merging methods can corrupt retained memories. Its fixed-budget design splits the cache into an exact main cache and compact residual entries, each representing a cluster with a mean key, mean value, and population count. Through a shared softmax, residual entries restore omitted attention mass inside the same normalization rather than applying a post-hoc output correction. A construction-time validation proxy allocates residual slots per layer and KV head, while a decode-time dynamic gate suppresses residual influence when the main-cache attention is sharply concentrated. Experiments on LLaMA-3.1-8B-Instruct and Qwen-2.5-7B-Instruct, using LongBench and RULER, show improvements under identical retained KV budgets: ResKV improves all 32 displayed LongBench configurations with an average gain of 1.02 points, and improves 63 of 64 displayed RULER configurations with an average gain of 3.38 points. In query-agnostic construction, the average RULER improvement reaches 4.54 points, compared with 2.22 points for query-aware construction, while tight LongBench budgets produce gains of 3.47 points at 10% retained KV and 3.66 points at 20% retained KV. The method preserves the compressed baseline’s peak memory footprint and stable long-context decoding, although its residual branch adds moderate throughput cost.