VidaForge: Open Research Infrastructure for Video Pretraining Data Recipes
AuthorsYan Ma, Jiadi Su, Zhulin Hu, Ethan Chern, Linhao Zhang, Tiantian Mi, Pengfei Liu
Resources
VidaForge makes video foundation-model data pipelines inspectable and reproducible, helping researchers test how dataset curation choices shape model performance.
Key results
Scene-level clips released in the open dataset
Total hours of released video
Downstream generation score for the broader-coverage recipe
Final Something-Something V2 top-1 accuracy for the mixed recipe
What the paper found
VidaForge is an open, attribution-ready infrastructure that turns raw video into model-specific pretraining data through five traceable stages: ingestion, segmentation, selection, annotation, and packaging. It preserves intermediate assets, recipe parameters, duplicate relations, quality scores, and sample-level decisions, so researchers can change filtering or segmentation rules and connect each resulting dataset variant to training outcomes without repeating reusable computation. Its selection pipeline combines optical, motion, aesthetic, and visible-text scoring with Meta’s PDQ perceptual deduplication and NVIDIA’s Cosmos-Embed semantic deduplication, while annotation uses Google’s Gemma-4-E4B-it and Qwen3.6-27B-FP8. The released VidaForge-3M dataset processes 800k source videos into 3.14M scene-level clips totaling 6,475.1 hours. In controlled from-scratch experiments using Wan 2.1-1.3B for generation and V-JEPA 2.1-1B for representation learning, a broader-coverage mixed recipe consistently outperformed higher-quality or lower-quality filtered recipes on downstream evaluation, even when pretraining loss preferred another dataset. Wan’s mixed recipe reached a VBench Total score of 60.64, compared with 59.74 for the selected high-quality recipe, while V-JEPA’s mixed recipe achieved 16.256% top-1 accuracy on Something-Something V2, versus 15.712% for Selected and 15.099% for Rejected. The central result is that lower training loss is not a reliable proxy for downstream video capability, and traceable data recipes make this divergence experimentally measurable.
Original abstract
Video foundation models increasingly rely on large-scale pretraining data, yet the end-to-end data pipelines behind them remain largely closed and difficult to inspect or reuse. Researchers seeking to understand how video data recipes affect model pretraining often need to build substantial infrastructure before testing even a focused hypothesis. We present VIDAFORGE, an open research infrastructure that represents a video data recipe as an executable five-stage workflow from raw videos to training datasets. A decision in this workflow can be varied to construct alternative datasets while preserving how every sample was produced. To demon strate this research workflow, we compare data recipes with different coverage and quality in early from-scratch pretraining of Wan 2.1 and V-JEPA 2.1. Across both learning objectives, the broader-coverage recipe achieves the highest downstream benchmark scores, while loss-based evaluation favors different recipes. This study demonstrates how VidaForge connects data-recipe choices to downstream model performance. We further release VIDAFORGE-3M, containing 3.14 million scene level clips totaling 6,475 hours, with fine-grained annotations and curation signals for video data-recipe research.
Read the original paperMore in Foundation Models
Browse all 47 papers →How Much Is an AI Token Worth? Scaling Laws for Wild AI-Generated Web Text
Jenna Russell, Ben Glickenhaus, Katherine Thai, John Wieting, Mohit Iyyer, Max Spero, Bradley Emi
AI-generated web text can help language models at first, but beyond a tipping point it degrades performance on human writing, making data filtering and separate evaluation increasingly important.
TabFM: A Zero-Shot Foundation Model for Tabular Data
Weihao Kong, Erez Louidor Ilan, Shuxin Nie, Taman Narayan, Rajat Sen, Yichen Zhou, Deqing Fu, Samet Oymak, Abhimanyu Das
TabFM is a large synthetic-data-trained model that aims to make accurate tabular predictions instantly, without retraining for each new dataset.
When Do Biological Reasoning Models Use Their Biological Inputs?
Ada Fang, Nikitha Thoduguli, Lukas Fesser, Hanlin Zhang, Sham M. Kakade, Marinka Zitnik
The study finds that many biological reasoning systems appear to succeed without meaningfully using the biological inputs they were designed to reason over.