Extreme Compression of Per-Layer Embeddings via Cross-Layer Latent Vector Quantization
AuthorsJiacheng Liu, Yaxin Luo, Xiaohan Zhao, Peng Sun, Xinyi Shang, Jiacheng Cui, Yonina C. Eldar, Zhiqiang Shen
AffiliationsMohamed bin Zayed University of Artificial Intelligence · Westlake University · Zhejiang University · University College London · Weizmann Institute of Science
A new way to compress Transformer embeddings dramatically reduces memory use while largely preserving model performance.
Key results
Seven-benchmark retention with OPTIC.
Seven-benchmark BF16 E4B retention at about 100× compression.
Seven-benchmark retention with OPTIC.
BF16 E4B seven-benchmark retention.
BF16 E4B seven-benchmark retention.
BF16 E4B compact inference with a fixed key-value cache.
What the paper found
Gemma 4’s Per-Layer Embeddings store a separate vector for each token at every decoder layer, creating a large compression target. OPTIC, or Output-Path-fitted, Token-shared Indexed Codes, addresses this by storing one quantized code tuple per token and reconstructing each layer’s vector with its own affine decoder. Rather than minimizing error in the stored embeddings, OPTIC fits codes and decoders to the gated PLE branch’s projection output, using alternating continuous updates and code reassignment. On Gemma 4 E4B, OPTIC preserves 97.8% of uncompressed BF16 performance across seven benchmarks at about 100× compression, compared with 66.3% for product quantization; on the official W4A16 model, it retains 96.9%. It retains 90.5% at 594.431× compression and 84.1% at 1,002.189×. The benchmarks include GSM8K, AIME 2026, and InfographicVQA. In direct inference tests using vLLM, compact BF16 PLE reduced GPU memory use by 29.7%, with at most 0.40% added wall time. These results show that sharing token codes across layers, while preserving layer-specific decoding and fitting for the model’s actual output path, can compress Gemma 4 embeddings far more effectively than conventional quantization.
Original abstract
Scaling embedding capacity alongside Transformer computation is an emerging trend in language models. This trend makes high compression rates increasingly useful. Hence, we study extreme compression of Per-Layer Embeddings (PLE), which follow this trend and account for 35.2% of the parameters of Gemma 4 E4B. At such rates, conventional quantizers give each token-layer vector its own code of only a few bytes. We instead develop OPTIC (Output-Path-fitted, Token-shared Indexed Codes), which stores one quantized code tuple per token and decodes it at every layer with a layer-specific affine decoder. Codes and decoders are fitted to the PLE output path rather than to the stored vectors. At about $100\times$ PLE compression of bfloat16 (BF16) E4B, OPTIC retains an aggregate 97.8% of the uncompressed model's performance across seven benchmarks, whereas product quantization, the strongest of six conventional baselines, retains 66.3%. At $594\times$ and $1,002\times$ PLE compression of BF16 E4B, OPTIC retains 90.5% and 84.1%. On the official E4B release with 4-bit linear weights and 16-bit activations (W4A16), OPTIC retains 96.9% at about $100\times$ PLE compression. During inference at about $100\times$ PLE compression, BF16 E4B uses 29.7% less GPU memory than with uncompressed PLE at the same key-value cache size, while running at most 0.40% slower.
Read the original paperMore in Efficient AI
Browse all 70 papers →AB-Attention: Accurate Block Attention for Accelerating Video Diffusion Transformers
Bin Xu, Mo Zhang, Zhengfeiyang Zhang, Xinfeng Yang, Yixing Ma, Wei Wang, HeXin Wang, Pengfei Zhou, Wangbo Zhao
AB-Attention speeds up video generation by skipping most attention computations while more accurately estimating the attention it omits.
Rethinking Sequence Compression with Test-time Regression
Sihan Yang, Jie Huang, Hanxuan Li, Yifan Guo, Weigao Sun, Yiran Zhong, Weiyang Liu
A new regression-based compressor aims to keep language and vision models efficient while preserving more useful information in fewer tokens.
Channel Selection for Mixture-of-Experts Decoding Needs No Calibration Data
Geunhyoung Kim, Sung Ju Hwang
A calibration-free way to skip most MoE expert-channel reads aims to make decoding faster while preserving model quality.