Shared Geometry As A Rosetta Stone: Cross-Modal Alignment Without Paired Data

This paper shows that independently trained models can often be aligned across modalities without paired examples, labels, or shared reference points. Its method, Wasserstein Procrustes with geometric initialization, repeatedly clusters each embedding space, matches cluster geometry using centered kernel alignment, then alternates sample matching with orthogonal-map updates. On MS COCO, across 21 vision–language model pairs, it achieves average FOSCTTM of 0.154, compared with 0.223 for mini-vec2vec; the alignment is coarse rather than an exact recovery of every image-caption match. With DINOv2 ViT-B/14 and Qwen3-8B, the method also transfers semantics to zero-shot classification, reaching 47.4% accuracy on CIFAR-10. Its performance depends on how similarly the two spaces organize data: across scientific and medical modality pairs, CKA explains 69% of alignment variance, with a Spearman correlation of −0.82. Adding a small number of known pairs improves results further: with up to 20 pairs, FOSCTTM is 14× to 28× lower than the strongest baseline. The authors also use the learned mapping to condition an image generator from text embeddings, producing broad scene and object categories without paired training data, though fine details remain less reliable. The central contribution is a practical geometric bridge for cross-modal tasks where paired data is scarce or unavailable.