Scaling Laws for Multimodal Data Mixtures
AuthorsAditi Khandelwal, Ayush K. Tarun, Yixuan Xu, Imanol Schlag, Golnoosh Farnadi, Siva Reddy, Luke Zettlemoyer, Antoine Bosselut
AffiliationsMILA - Quebec AI Institute, McGill University · EPFL · ETH · University of Washington
A large-scale study finds that text, vision, and audio need different data and model-size tradeoffs—and that vision can hinder audio learning.
Key results
Controlled MoE pretraining experiments used to fit the scaling law.
Largest token budget per training run.
Audio compute allocation favors model capacity over token growth.
Indicates strong interference from vision when predicting audio loss.
Eight of nine modality-loss predictions fall within this error.
What the paper found
As systems such as Google Gemini, Meta Llama, and OpenAI GPT-4o move toward native multimodality, this study offers a way to predict how text, vision, and audio mixtures affect training loss in Mixture-of-Experts models. Its scaling law combines modality-specific compute needs with cross-modal transfer, capacity sharing, and diminishing returns from repeated data. The researchers fit it across 268 experiments, scaling models to 8.3B parameters and training runs to 150B tokens, using curated mixtures informed by Apertus, Chameleon, and Voxtral. Text and vision favor balanced growth in model size and data, while audio is more model-heavy: its compute-optimal model-size exponent is 0.62. Interactions are asymmetric: text and vision interfere with each other, and vision is particularly disruptive to audio, with a vision-to-audio transfer coefficient of -0.795; audio has no reliable effect on text or vision. Repeated vision and audio data lose value quickly, with little benefit beyond two passes compared with adding unique data or model capacity. On held-out larger-scale and continual-pretraining runs, 8 of 9 modality-loss predictions fall within 2.6% of observed values. The findings provide a practical, context-specific recipe for allocating multimodal training compute, rather than a universal mixture prescription.
Original abstract
Frontier AI systems are increasingly natively multimodal, jointly pretrained on multiple modalities, such as text, vision, and audio. However, the scientific understanding of the multimodal data mixtures used to pretrain these systems --- how much of each modality to mix, and at what potential interference cost to the others --- is largely absent or gatekept. In this work, we derive data mixing scaling laws for three modalities: text, vision, and audio. Multimodal Large Language Models (MLLMs) increasingly adopt Mixture-of-Experts (MoE) architectures, as MoEs scale efficiently and partially limit cross-modal interference. We therefore use MoEs to conduct 268 experiments ranging from 457M to 8.3B parameters, on up to 150 billion tokens. Our work leads to multiple actionable insights into multimodal data mixing for training MLLMs. First, compute optima are modality-specific: audio modality is model-heavy, favoring scaling parameters over tokens, while text and vision require balanced scaling of model size and data. Second, repeating vision or audio data beyond 2x yields negligible benefits. Finally, cross-modal interactions are highly asymmetric: vision strongly interferes with audio, whereas audio's interference with learning text and vision is negligible. Overall, we provide a principled foundation for understanding the multimodal data mixtures needed to train frontier MLLMs.
Read the original paperMore in Multimodal AI
Browse all 67 papers →Rethinking Projector Training in Multimodal LLMs
Fabian Gröger, Naël Ouerghemi, Shuo Wen, Maria Brbić
The paper shows that multimodal LLMs may not need projector training at all, introducing a data-free method that nearly matches the standard approach.
Shared Geometry As A Rosetta Stone: Cross-Modal Alignment Without Paired Data
Dominik Schnaus, Thomas Dagès, Daniel Cremers, Xi Wang, Phillip Isola
The work aligns independently trained image and text representations—even without paired examples—by exploiting geometry they may already share.
ALIVE: Interaction-Aligned Object Insertion for First-Frame-Guided Video Editing
Zhenghong Zhou, Zhe Lin, Jiebo Luo, Yuqian Zhou
ALIVE makes inserted video objects behave as if they truly belong in the scene by learning how they should interact with existing actions.