DataComp-VLM: Improved Open Datasets for Vision-Language Models

DataComp-VLM, from collaborators across Google, Google DeepMind, Stanford University, LAION, and the Tübingen AI Center, reframes vision-language model training as a controlled data-curation problem and shows that mixing strategy dominates filtering. The benchmark, DCVLM, aggregates 160 public datasets into a 6T-token multimodal pool spanning image-caption pairs, interleaved documents, text-only corpora, and multimodal instruction data, then evaluates models from 1B to 8B parameters on up to 52 benchmarks across 9 domains. Across more than 1,000 experiments, common quality filters such as CLIP-score, text-quality classifiers, and multimodal perplexity give little or no gain once the pool is already curated, with the best filtering result only +0.8pp. In contrast, an instruction-heavy recipe of 10% image-caption, 5% multimodal documents, 15% text-only, and 70% instruction-tuning data scales best, and pretraining rankings transfer to supervised fine-tuning with Pearson r=0.99 across 54 SFT runs. The resulting DCVLM-BASELINE trains an 8B model on 200B tokens to 63.6% on the 33-task Core suite, beating FineVision by +5.4pp, while a 4B model trained on 100B tokens already surpasses an 8B FineVision model trained on 200B tokens, implying a 4× compute reduction. The paper’s main takeaway is that, for modern autoregressive VLMs, data composition is a stronger lever than post hoc filtering, especially at larger scales.