Rethinking Sequence Compression with Test-time Regression
AuthorsSihan Yang, Jie Huang, Hanxuan Li, Yifan Guo, Weigao Sun, Yiran Zhong, Weiyang Liu
AffiliationsThe Chinese University of Hong Kong · Independent Researcher · Token Foundry, Alibaba Group
A new regression-based compressor aims to keep language and vision models efficient while preserving more useful information in fewer tokens.
Key results
Convolutional TTR-Compressor average, compared with 0.3920 for the strongest conventional compressor.
TTR-Compressor result for a 1B-parameter model trained on 100B tokens; attention-weighted pooling scored 0.735.
Convolutional TTR-Compressor score across multimodal benchmarks.
Convolutional TTR-Compressor score, close to the MLP merger’s 54.93 with 2×2 merging.
Fewer visual tokens with 3×3 merging than the 2×2 configuration.
What the paper found
TTR-Compressor replaces fixed pooling and projections with test-time ridge regression: each token block defines a regression problem, and its fitted coefficients become the compressed representation. The regression targets and diagonal regularizer are learned end-to-end, while inference computes an exact dual solution with a T×T Cholesky solve. Because the solve uses the tokens’ Gram matrix, it models inter-token correlations and reduces the influence of redundant evidence. On RULER, its convolutional variant reached 0.5144 average retrieval accuracy, versus 0.3920 for the strongest conventional compressor; at larger scale, it scored 0.7996 versus 0.735 for attention-weighted pooling using a 1B-parameter model trained on 100B tokens. For visual compression with Qwen3-VL and Qwen3-4B-Instruct, the convolutional method averaged 56.03 on multimodal benchmarks with 2×2 merging. With 3×3 merging, it scored 54.85—close to the standard MLP merger’s 54.93 at 2×2—while using 55.6% fewer visual tokens. The findings position regression-based, input-adaptive compression as a practical alternative for long-context language models and multimodal systems, including model families such as DeepSeek and Qwen.
Original abstract
Sequence compression is a fundamental operation and a critical information bottleneck in efficient language and multimodal models, and yet prevailing approaches based on pooling or learned projections remain underperforming. We revisit sequence compression through the lens of test-time regression: input tokens define a regression problem whose fitted coefficients form the compressed representation. Importantly, the regression problem is learned end-to-end via backpropagation, while its coefficients are computed efficiently in closed form during the forward pass. This leads to TTR-Compressor, a general alternative to conventional pooling and projection-based compression. We evaluate TTR-Compressor as the compressor for block-wise sparse attention and the visual token merger for multimodal large language models. On block-wise sparse attention, TTR-Compressor improves RULER performance over attention-based compressors by a large margin. For visual token merging, it consistently outperforms the standard MLP merger and matches its performance with 55.6% fewer visual tokens. These results establish test-time regression as an effective primitive for sequence compression in efficient language and multimodal models.
Read the original paperMore in Efficient AI
Browse all 70 papers →AB-Attention: Accurate Block Attention for Accelerating Video Diffusion Transformers
Bin Xu, Mo Zhang, Zhengfeiyang Zhang, Xinfeng Yang, Yixing Ma, Wei Wang, HeXin Wang, Pengfei Zhou, Wangbo Zhao
AB-Attention speeds up video generation by skipping most attention computations while more accurately estimating the attention it omits.
Extreme Compression of Per-Layer Embeddings via Cross-Layer Latent Vector Quantization
Jiacheng Liu, Yaxin Luo, Xiaohan Zhao, Peng Sun, Xinyi Shang, Jiacheng Cui, Yonina C. Eldar, Zhiqiang Shen
A new way to compress Transformer embeddings dramatically reduces memory use while largely preserving model performance.
Channel Selection for Mixture-of-Experts Decoding Needs No Calibration Data
Geunhyoung Kim, Sung Ju Hwang
A calibration-free way to skip most MoE expert-channel reads aims to make decoding faster while preserving model quality.