Youtu-Parsing-Omni: One Encoder, One Schema, All Modalities
AuthorsYunhang Shen, Zhangquan Chen, Xiaotian Li, Zhongpeng Cai, Qing Zhou, Zekai Zhang, Kun Zhang, Zezhong Wu, Jian Hu, Haihan Gao, Heting Gao, Xin Li, Yinsong Liu, Deqian Jiang, Di Yin, Haoyu Cao, Xing Sun, Bing Liu, Yanqiu Qu, Yunfei Wu, Kun Yin, Zhixiang Wei, Yongxiang Hua, Qianyu Li, Xiaoxing Hu, Peiyuan Zhang, Lijiang Li, Dong Xu, Zhenchuan Yang, Yiliang Jiang
Youtu-Parsing-Omni uses one model and one output schema to turn documents, images, audio, and video into structured, evidence-grounded records.
Key results
One checkpoint handles the paper’s multimodal parsing tasks.
Tasks unified under OmniSchema.
Youtu-Parsing-Omni’s score.
Highest score among open-weight models.
OmniDocBench points added by six OSAD rounds.
Fraction of the total post-SFT OmniDocBench improvement attributed to OSAD.
What the paper found
Youtu-Parsing-Omni turns documents, images, audio, and video into structured records that preserve where and when evidence appears. Its central design is a text-initialized shared Transformer encoder: visual and audio tokens use a common space-time coordinate system and can attend to one another in local video windows. A single OmniSchema defines grounded outputs for 10 tasks, while OmniSchema-Aware On-Policy Distillation, or OSAD, trains on the model’s own parses with token-level supervision: Jensen–Shannon divergence for content and forward KL divergence for schema structure, with the teacher refreshed as training progresses. One 5 B-parameter model scores 96.96 on OmniDocBench v1.6 and leads open-weight models on OmniParsingBench with 75.08; Gemini-3-Pro scores 77.44 overall there. Six OSAD rounds contribute 1.75 points, or 88 percent of the post-SFT gain on OmniDocBench. The model also shows strong specialist-level results in chemical and music transcription, areas where systems such as DeepSeek-OCR 2 and dedicated optical-music models provide useful comparisons.
Original abstract
We present Youtu-Parsing-Omni, an omni-modal parser that converts documents, images, audio, and video into structured records grounded in space and time, providing a perception layer for agents that read, locate, and cite evidence across modalities. Its design rests on one encoder for every modality and one schema for every task. Youtu-Omni-Encoder, initialized from a pre-trained language model to strengthen text reading, encodes all modalities with shared weights in a unified space-time coordinate system, and a few fusion layers let video frames attend to concurrent audio, bringing cross-modal fusion and temporal alignment into perception itself. OmniSchema unifies the outputs of all ten parsing tasks, allowing a single self-evolving post-training procedure, OmniSchema-Aware On-Policy Distillation (OSAD), to provide a shared token-role-based objective across tasks. In each round, the model is trained on its own rollouts against token-level targets from a reference-conditioned copy of itself, which is then refreshed with the trained weights, so that the parser and its supervision co-evolve. With a single 5 B-parameter checkpoint, Youtu-Parsing-Omni achieves state-of-the-art performance on OmniDocBench v1.6 (96.96 Overall) and leads all open-weight models on OmniParsingBench (75.08 Avg.), with its advantage concentrated on audio and video. It also achieves the best overall chart-parsing score (95.02) and the highest exact chemical-fingerprint agreement among compared systems, and matches a dedicated optical-music model in character-level transcription of typeset sheet music while staying far ahead of general-purpose systems. Code and model weights are publicly available.
Read the original paperMore in Multimodal AI
Browse all 70 papers →Generalized CycleGRPO for Fine-Grained Multimodal Understanding and Grounding
Xin Zhang, Haochen Wang, Beibei Lin, Zhaoxiang Zhang, Robby T. Tan, Jason Li, Dacheng Tao
A new training framework teaches multimodal models to describe a visual region and then find that same region again.
Proving and Improving the Platonic Representation Hypothesis with JEPAs
Randall Balestriero, Yann LeCun
The paper argues that training on meaningful cross-modal views aligns representations far more efficiently than relying on next-token prediction and scale alone.
Rethinking Projector Training in Multimodal LLMs
Fabian Gröger, Naël Ouerghemi, Shuo Wen, Maria Brbić
The paper shows that multimodal LLMs may not need projector training at all, introducing a data-free method that nearly matches the standard approach.