RankE: End-to-End Post-Training for Discrete Text-to-Image Generation with Decoder Co-Evolution

RankE, from Westlake University, Zhejiang University, Tsinghua University, Hong Kong University of Science and Technology, and Shanghai AI Lab, is a first end-to-end post-training framework for discrete text-to-image generation that targets a failure mode the paper calls Latent Covariate Shift: reward optimization improves the autoregressive policy while the frozen VQ decoder drifts out of its training distribution and image fidelity collapses. The method alternates between GRPO-based policy updates and decoder adaptation, using a ranking objective at two levels: token-level group-relative advantages for the policy and pixel-level reward-weighted Rank-GAN plus direct reward backpropagation for the decoder, while anchoring the decoder with reconstruction and EMA-consistency regularization. On LlamaGen-XL at 775M parameters, RankE breaks the usual CLIP–FID trade-off on MS-COCO 30K, reaching 33.76 CLIP and 15.21 FID, versus standard RL’s 32.45 CLIP and 17.76 FID; on Janus-Pro-1B it also improves CLIP to 33.86 and lowers FID to 25.19. The mechanism is supported by diagnostics showing standard RL widens token-distribution KL divergence by 24% and reduces codebook entropy, while RankE keeps both near the SFT regime. Ablations confirm that removing Rank-GAN, reconstruction, consistency, or reward backpropagation degrades performance, and the full method is trained on 8× NVIDIA A100 GPUs with a 16,384-entry codebook over a 15K-caption corpus.