Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?

This paper tests whether cross-lingual alignment requires multilingual joint training by comparing strictly monolingual Transformer models trained on disjoint corpora, vocabularies, and parameters. Across Goldfish models evaluated with FLORES-200, Tatoeba, OPUS, and BouQUET, position-weighted SGPT representations produced matched last-layer CKA of 0.78 on FLORES-200 versus 0.17 for shuffled sentence pairs, showing semantic geometry beyond architectural or token-level overlap. Alignment increased with training-data scale, model scale, and linguistic proximity, and remained detectable across five independently developed approximately 1B monolingual models spanning Pythia, Llama, Qwen2.5, and Mistral-based systems, where matched CKA averaged 0.71 versus 0.18 shuffled. For construction, a single orthogonal Procrustes rotation learned from parallel sentences achieved 88.7% cosine-retrieval accuracy, outperforming affine and one-layer MLP mappings because it preserves angular geometry. For causation, cross-model activation patching injected rotated English residuals into German, French, Spanish, Japanese, and Chinese models; on country-to-capital factual cloze prompts, the transferred representation achieved up to 85% directional success, compared with 76–98% within-model ceilings and approximately 50% chance controls. The findings support the Platonic Representation Hypothesis: language models can independently converge toward partially universal conceptual representations, enabling post-hoc model stitching, merging, and modular multilingual systems without joint pretraining.