Generalized CycleGRPO for Fine-Grained Multimodal Understanding and Grounding
AuthorsXin Zhang, Haochen Wang, Beibei Lin, Zhaoxiang Zhang, Robby T. Tan, Jason Li, Dacheng Tao
A new training framework teaches multimodal models to describe a visual region and then find that same region again.
Key results
SAMTok improves from 57.5% to 68.2%.
Qwen3-VL improves from 14.8% to 33.3%.
Qwen3-VL improves from 45.6%.
Human-verified contrastive video/query groups.
What the paper found
Generalized CycleGRPO trains one multimodal model to describe a specified region and then localize that description back to its source. Group-relative policy optimization rewards descriptions when the reconstructed region overlaps the original, and separately improves localization trajectories. The same region–text–region cycle works with image masks or boxes, video intervals, and timestamped spatial evidence, using region annotations without reference captions or an external language-model judge during reinforcement learning. To test whether video captions identify a particular event rather than a similar recurrence, the paper introduces DOVE, a contrastive benchmark with 180 groups and 530 target intervals; Gemini 3 Pro helps propose annotations that are then human-verified. Results include a 10.7-percentage-point gIoU gain for SAMTok on GroundingSuite, from 57.5% to 68.2%; on DOVE, Qwen3-VL’s Temporal Evidence Coverage rises from 14.8% to 33.3%, a gain of 18.5 percentage points. Qwen3-VL’s Charades tIoU improves from 45.6% to 51.9%, while Open-o3-Video’s V-STaR mLGM increases from 47.2% to 49.8%. Together, these findings show that reconstruction can provide a shared, geometrically verifiable learning signal for fine-grained understanding and localization across images and video.
Original abstract
Fine-grained multimodal intelligence requires models to both understand a designated region and localize it from language. Existing methods learn these directions with separate objectives and task-specific annotations. We present Generalized CycleGRPO, the first unified reinforcement learning framework coupling region understanding and localization across spatial, temporal, and spatio-temporal domains. Given a region, a multimodal large language model generates candidate descriptions and then localizes each description to reconstruct the input region. Reconstruction overlap provides a verifiable reward for captioning, while individual localization rewards improve the inverse mapping. A common region-space formulation extends this cycle from image masks and boxes to video intervals and timestamped spatial evidence, without reference captions or an external language-model judge during reinforcement learning. In videos, the cycle encourages descriptions that distinguish a target event from similar recurrences. We introduce DOVE (Distinctive Occurrence-level Video-caption Evaluation), a human-verified contrastive benchmark for this temporal distinctiveness. CycleGRPO improves SAMTok's overall gIoU from 57.5% to 68.2% on GroundingSuite and average gIoU from 71.2% to 78.7% on GRES. For temporal tasks, it achieves 51.9% tIoU on Charades and 33.3% Temporal Evidence Coverage on DOVE, compared with 45.6% and 14.8% for the Qwen3-VL initialization. These results establish region-text cycle consistency as a general self-evaluation principle for multimodal understanding and localization. Models, code, and DOVE are available at https://devinxzhang.github.io/Generalized-CycleGRPO-Page/.
Read the original paperMore in Multimodal AI
Browse all 70 papers →Proving and Improving the Platonic Representation Hypothesis with JEPAs
Randall Balestriero, Yann LeCun
The paper argues that training on meaningful cross-modal views aligns representations far more efficiently than relying on next-token prediction and scale alone.
Youtu-Parsing-Omni: One Encoder, One Schema, All Modalities
Yunhang Shen, Zhangquan Chen, Xiaotian Li, Zhongpeng Cai, Qing Zhou, Zekai Zhang, Kun Zhang, Zezhong Wu, Jian Hu, Haihan Gao, Heting Gao, Xin Li, Yinsong Liu, Deqian Jiang, Di Yin, Haoyu Cao, Xing Sun, Bing Liu, Yanqiu Qu, Yunfei Wu, Kun Yin, Zhixiang Wei, Yongxiang Hua, Qianyu Li, Xiaoxing Hu, Peiyuan Zhang, Lijiang Li, Dong Xu, Zhenchuan Yang, Yiliang Jiang
Youtu-Parsing-Omni uses one model and one output schema to turn documents, images, audio, and video into structured, evidence-grounded records.
Rethinking Projector Training in Multimodal LLMs
Fabian Gröger, Naël Ouerghemi, Shuo Wen, Maria Brbić
The paper shows that multimodal LLMs may not need projector training at all, introducing a data-free method that nearly matches the standard approach.