Where Does Alignment Emerge and What Does It Mean? A Closer Look at the Platonic Representation Hypothesis
AuthorsJulia Herbst, Simon Roschmann, Quentin Bouniot, Zeynep Akata
AffiliationsHelmholtz Munich · Technical University of Munich · Munich Center for Machine Learning · LTCI, Télécom Paris, Institut Polytechnique de Paris, Palaiseau, France
The study finds that language and vision models can develop surprisingly similar representations, but still fall short of the alignment seen between models trained on the same kind of data.
Key results
Mutual k-NN score between DINOv3 and Qwen3-8B at their maximally aligned layers.
Approximate number of language representation dimensions that captures much of the cross-modal alignment.
Pearson correlation between MMLU accuracy and alignment on detailed DOCCI captions.
Pearson correlation between ARC Challenge accuracy and alignment.
Approximate mutual k-NN alignment between models operating within the same modality.
What the paper found
This study takes a layer-by-layer look at the Platonic Representation Hypothesis: whether vision and language models develop similar representations of the world. Comparing vision encoders with 21 generative language models—including Meta’s Llama, DeepSeek, Qwen, Mistral, and Gemma—using mutual k-nearest-neighbor overlap on WIT, CC3M, COCO, and DOCCI, the researchers find that cross-modal alignment mostly appears in later layers. Its peak on WIT is 0.143 between DINOv3 and Qwen3-8B, while principal-component analysis shows that alignment can largely be retained in about 100 dimensions, despite language representations spanning 1,536–5,120 dimensions. The link between language-model performance and alignment depends on caption quality: on detailed DOCCI captions, alignment correlates with MMLU accuracy at Pearson r = 0.66, and with ARC Challenge accuracy at r = 0.73, but does not consistently track GSM8K math performance. For context, unimodal vision-language pairs exceed 0.3 alignment, outperforming CLIP and SigLIP 2 on COCO and DOCCI, but remaining below Qwen3-VL-Embedding-8B and the roughly 0.5 alignment seen between models of the same modality. The results suggest that cross-modal alignment captures some shared semantic structure, not a universal measure of model capability.
Original abstract
The Platonic Representation Hypothesis (PRH) suggests that neural networks are converging toward similar representation spaces. As language models become more capable, their representations are aligning with those of vision models. However, recent work has questioned whether this trend continues to hold for newer language models. Moreover, it has been shown that representation similarity metrics can be confounded by model depth and width, requiring appropriate baselines for interpretation. In this work, we first investigate where alignment emerges between unimodal models across model layers and representation dimensions. Next, we find that the PRH capability-alignment trend persists for newer models as long as the captions are sufficiently clean and informative. Finally, we contextualize cross-modal alignment, introducing unimodal and multimodal baselines on various datasets. Our results show that the cross-modal representational alignment of unimodal models can surpass that of explicitly aligned vision-language models, but remains substantially behind the alignment observed for models of the same modality. Code is available at https://github.com/ExplainableML/prh-closer-look.
Read the original paperMore in Multimodal AI
Browse all 79 papers →ALIVE: Making Streaming Video Models Interactive
Jinzhao Li, Yuan Xu, Jinxuan Wu, Yukun Wang, Yijia Lei, Hanyu Ji, Yinuo Chen, Zining Huang, Chi Zhang, Hang Ye, Yiwu Zhong, Honglei Yan, Panwang Pan, Yizhou Wang, Yin Li, Miao Liu
ALIVE aims to make video-streaming AI an interactive partner that can monitor events, coordinate tasks, and ask for feedback.
Let Samples Sing in Chorus: Mitigating Spurious Correlations via Cross-view Consensus
Weiwei Li, Junzhuo Liu, Tong Chu, Hengfu Yu, Yinjie Lei, Wen Li
CHORD combines clues from multiple representation spaces to help models rely less on misleading shortcuts and perform better on underrepresented groups.
LightTAR: Light and Fast Target-Aware Representations for Multimodal Tabular Learning
Eilam Shapira, Alan Arazi, Shoham Grunblat, Roi Reichart
LightTAR adds a small set of target-aware features to standard embeddings, helping tabular models make better predictions with little extra runtime.