Moving Alphabet: A Controlled Study of Training Data for Text-to-Video Generation

Researchers from Meta Superintelligence Labs and Purdue University introduce Moving Alphabet, a procedural testbed that renders one to three moving letters with controllable appearance, motion, duration, and caption corruption, isolating training-data effects that are difficult to study in real video datasets used by systems such as OpenAI’s Sora. They train an 800M-parameter MMDiT flow-matching model with T5-XXL and CLIP ViT-L/14 text encoders, using 150K videos per baseline experiment. Balanced mixtures consistently outperform specialist datasets: equal mixing of 1-, 2-, and 3-letter scenes reaches 12.9 dB FG PSNR on 3-letter evaluation, versus 10.1 dB for pure 3-letter training, while equal mixing of 2-, 4-, and 8-second clips achieves 12.2 dB average FG PSNR with fewer tokens than long-video-only training. Caption correctness is especially damaging when degraded: FG PSNR falls from 12.3 to 4.3 dB as precision drops from 1.0 to 0.3, and corrupted captions can require up to 4 times the compute to reach the same validation loss. Classifier-free guidance improves categorical prompt adherence for moderate corruption but cannot restore pixel fidelity; fine-tuning on clean data recovers at most 55% of lost FG PSNR in the reported moderate-corruption case, and severe caption errors remain largely irreversible. The central recommendation is to prioritize balanced data distributions and accurate captions during pre-training rather than relying on inference-time or post-training fixes.