Channel-wise Vector Quantization

Channel-wise Vector Quantization, from Shanghai Innovation Institute, Westlake University, Zhejiang University, Fudan University, JD.COM, and the University of Chinese Academy of Sciences, replaces patch-wise image tokens with channel-wise tokens: each latent channel becomes one discrete codeword, and autoregressive generation is reformulated as next-channel prediction rather than next-patch prediction. The key result is structural, not incremental: CVQ reaches 100% codebook utilization with a 16,384-entry codebook, while conventional VQ collapses to 4.5% utilization at the same size. On ImageNet-1K reconstruction, vanilla CVQ improves rFID to 2.60 at 256 tokens and 0.88 at 1024 tokens, with PSNR rising to 25.02 dB and SSIM to 0.723. The paper also shows that scaling the codebook to 65,536 entries preserves 96.1% utilization and reduces rFID to 2.32, a 52% reconstruction improvement over the VQ baseline. Built on CVQ, the channel-wise autoregressive model CAR is trained on 80M text-image pairs and a Qwen3-4B/8B backbone; the 8B version reaches GenEval 0.79 and DPG 86.72, while the 4B version scores 0.75 GenEval and 83.82 DPG. A simple nested channel-dropout scheme adds a coarse-to-fine ordering and boosts generation by 0.12 GenEval and 9.38 DPG without hurting reconstruction, indicating that channel tokens can support both efficient compression and strong text-to-image synthesis.