Proving and Improving the Platonic Representation Hypothesis with JEPAs
AuthorsRandall Balestriero, Yann LeCun
AffiliationsBrown University · AMI LabsCorrespondence: rbalestr@brown.edu Code Live demo · New York University
The paper argues that training on meaningful cross-modal views aligns representations far more efficiently than relying on next-token prediction and scale alone.
Key results
Good-view JEPA reaches the alignment achieved by next-token prediction using 42 times fewer training samples.
The model reaches leading cross-modal alignment on WIT-1024.
Scale of the web images whose associated texts were used in the largest training runs.
Mutual 10-nearest-neighbor alignment with DINOv2.
Zero-shot accuracy after alignment from unpaired data.
Captioning score using independently trained image and text encoders joined by a fitted linear map.
What the paper found
This paper turns the Platonic Representation Hypothesis—the idea that models of different modalities can learn a shared representation of the world—into a theorem for Joint-Embedding Predictive Architectures, or JEPAs. Under the theorem’s assumptions, separately trained encoders learn the same representation up to a rotation when their positive pairs link observations of the same scene. The key practical distinction is between “good views,” such as two captions of one image, and “bad views,” such as two masked versions of one caption. The authors show that next-token prediction behaves like a bad-view JEPA: a document prefix and its continuation often share wording, not the underlying scene. Across data ranging from 145k captions to texts associated with 940M web images, good-view training reaches the same alignment as next-token prediction with 42 times fewer samples. A 109M-parameter text JEPA scores .184 on WIT-1024 against DINOv2, exceeding the language models tested, including Alibaba’s Qwen2.5 and Meta’s Llama-3 models up to 72B parameters, as well as CLIP’s text encoder. In pair-free zero-shot evaluation, it reaches 64.4% top-5 accuracy on ImageNet-100, above the reported maximum of 56% for language-model methods up to 8B parameters. As a demonstration, a text-only-trained decoder combined with frozen DINOv2 and DINOv3 image encoders reaches .838 CIDEr on COCO; only a linear map between the independently trained encoders is fitted on image-caption pairs. The central finding is that choosing positive views that preserve shared scene meaning can align modalities more efficiently than scaling next-token prediction alone.
Original abstract
Whether models trained on different modalities converge to the same representation of the world is central to AI. The Platonic Representation Hypothesis (PRH) conjectures that they do as they scale, but without a proof it offers neither a guarantee nor a recipe. We turn the hypothesis into a theorem. First, we prove a \emph{Platonic Representation Theorem}: Joint-Embedding Predictive Architectures (JEPAs) trained separately on any number of modalities learn the same representation up to one rotation whenever their positive pairs are two observations of one scene--\emph{good views}--and their misalignment is bounded otherwise. Second, we show that next-token prediction is a JEPA with \emph{bad views}, parts of a single document, which explains why language models align with vision only through scale. Third, we validate the theory from 145k captions to the texts of 940M web images, and uncover a scaling law of alignment that predicts unseen runs within seed noise. Good views need 42 times fewer training samples than next-token prediction on the same corpus to reach the same alignment, and a 109M-parameter text JEPA aligns with DINOv2 and CLIP's image encoders better than every language model we evaluated, up to 72B parameters, and than CLIP's text encoder: next-token prediction is a remarkably inefficient route to platonic representations. As a showcase, one rotation turns separately trained encoders into an image captioner (CIDEr .838 on COCO). Even that rotation needs no pair: on the latest pair-free benchmark, our text JEPA beats every zero-shot accuracy reported with language models up to 8B parameters, e.g., 64\% against at most 56\% top-5 on ImageNet-100. Positive views, and not scale alone, are thus the lever to build platonic representations.
Read the original paperMore in Multimodal AI
Browse all 70 papers →Generalized CycleGRPO for Fine-Grained Multimodal Understanding and Grounding
Xin Zhang, Haochen Wang, Beibei Lin, Zhaoxiang Zhang, Robby T. Tan, Jason Li, Dacheng Tao
A new training framework teaches multimodal models to describe a visual region and then find that same region again.
Youtu-Parsing-Omni: One Encoder, One Schema, All Modalities
Yunhang Shen, Zhangquan Chen, Xiaotian Li, Zhongpeng Cai, Qing Zhou, Zekai Zhang, Kun Zhang, Zezhong Wu, Jian Hu, Haihan Gao, Heting Gao, Xin Li, Yinsong Liu, Deqian Jiang, Di Yin, Haoyu Cao, Xing Sun, Bing Liu, Yanqiu Qu, Yunfei Wu, Kun Yin, Zhixiang Wei, Yongxiang Hua, Qianyu Li, Xiaoxing Hu, Peiyuan Zhang, Lijiang Li, Dong Xu, Zhenchuan Yang, Yiliang Jiang
Youtu-Parsing-Omni uses one model and one output schema to turn documents, images, audio, and video into structured, evidence-grounded records.
Rethinking Projector Training in Multimodal LLMs
Fabian Gröger, Naël Ouerghemi, Shuo Wen, Maria Brbić
The paper shows that multimodal LLMs may not need projector training at all, introducing a data-free method that nearly matches the standard approach.