Understanding Data Temporality Impact on Large Language Models Pre-training

Understanding Data Temporality Impact on Large Language Models Pre-training, by Romain Fabre, Hippolyte Pilchen, Franck Signe Talla, Patrick Perez, and Edouard Grave at Kyutai, argues that standard shuffled pre-training obscures when facts were learned and produces a measurable “knowledge horizon” lag. The authors pretrain matched 6B-parameter Transformer decoders on 2.5T Common Crawl tokens in two regimes: a globally shuffled baseline and a strictly chronological curriculum spanning 2018–2025, using AdamW, a 4096-token context, and 128 H100 GPUs. To evaluate temporal grounding, they introduce KairosQA, a 7,167-pair benchmark built from Wikidata subject–relation–year triplets, filtered for temporal variation, popularity, and ambiguity, then tested in both cloze and generative settings; they also compare against TAQA. On OLMES, sequential training matches shuffled training on general language understanding, reaching comparable final scores while changing the learning trajectory. On KairosQA, however, chronological ordering consistently yields fresher factual knowledge: sequential checkpoints peak in the year immediately before their cutoff and outperform shuffled models on recent facts, reversing the recency decay seen in open-source baselines such as Llama 3.1-8B, Gemma 3, Qwen3, and Olmo3. The effect is strongest on 2023–2024 questions, where shuffled models often approach random accuracy, while the sequential model retains a clear margin. The paper also shows that simple replay or model soup only partially mitigates catastrophic forgetting, indicating that temporal ordering improves factual freshness but does not fully solve historical retention.