Training Foundation Models with a Single Algorithm

Foundation-model pipelines—including those associated with DeepSeek and OpenAI’s GPT-4—typically separate pretraining, midtraining, supervised fine-tuning, and reinforcement learning into distinct stages. This paper proposes one weighted maximum-likelihood objective for all four: RL becomes likelihood fitting to reward-selected generations, with MaxRL, REINFORCE, and continuous-reward TailRL represented through different target and prompt distributions. A shared asynchronous trainer mixes static text and demonstrations with generated responses, so training changes which data sources are sampled and in what proportions, not the learning algorithm. Experiments on a 124M-parameter reasoning model and a 0.76B-parameter real-world model find that mixing can improve reasoning, retention, and later adaptation, though outcomes depend on the data schedule and metric. After mathematical training with a 5% midtraining-data mixture, followed by 4000 coding-RL updates, the unified model reached 24.96% HumanEval+ pass@1, versus 22.94% for staged training, and 58.47% MBPP+ pass@16, versus 52.12%. In a separate implementation comparison, the asynchronous trainer’s update throughput was 7.8-fold higher than staged RL; this reflects system differences, not an isolated algorithmic speedup.