Training Stability under Adaptive Curricula
AuthorsJohan Obando-Ceron, Walter Mayor, Michael Dennis, Aaron Courville, Pablo Samuel Castro
AffiliationsMila – Québec AI Institute · Université de Montréal · Google DeepMind · CIFAR AI Chair
Adaptive training environments can undermine an RL agent’s ability to learn, so stable learners are essential for lasting performance.
Key results
Environment interactions used in standard experiments.
Seeds used for standard experiments.
Interactions in the long-horizon PLR experiment.
Parallel environments where stabilization still improved performance.
Clipping range scale used in the selected stabilization setup.
Categorical value support resolution in the selected stabilization setup.
What the paper found
A study involving Google DeepMind finds that adaptive reinforcement-learning curricula can keep selecting challenging environments even as the agent loses the ability to learn from them. In Kinetix, PPO agents trained with Prioritized Level Replay showed parameter growth, weaker ranking of outcomes by predicted value, and less diverse representations, especially on complex tasks. The proposed stabilization combines weight clipping with HL-Gauss, which trains value predictions as categorical distributions. Using a clipping factor of 2 and 51 value bins, the combination improved learning without changing curriculum rules or interaction budgets, and also helped with domain randomization, SFL, and ACCEL. The standard experiments used 2B environment interactions and 5 seeds; with training extended to 5B interactions, the stabilized agent continued improving while the baseline degraded. Stabilization also helped under limited rollout parallelism, including 128 environments, and supported zero-shot transfer and retention after task-distribution shifts. The central point is that curriculum quality depends not only on which tasks are selected, but also on whether the learner remains trainable on them.
Original abstract
Unsupervised Environment Design (UED) approaches train reinforcement learning (RL) agents on diverse curricula that may evolve with their capabilities. While prior work has largely focused on how these curricula are constructed, we study whether their students remain trainable under the resulting distributions. Across multiple UED mechanisms, environment complexities, and extended training horizons, we identify recurring learner-side pathologies, including parameter growth, degraded value--outcome ordering, and loss of representational diversity, that accompany stagnation and performance degradation. These findings connect limitations in UED to optimization failures previously identified in conventional deep RL, while showing how adaptive curricula couple the learner's optimization state to its future training distribution. Controlled interventions show that mitigating selected pathologies improves learning. Stabilizing the learner improves long-horizon performance and zero-shot generalization without changing the curriculum rule or increasing environment interaction. Our results establish learner trainability as a complementary design principle for UED. Effective curricula require students that remain capable of learning from the distributions they produce.
Read the original paperMore in Reinforcement Learning
Browse all 69 papers →AttnSR: Attention Step Reward for Long-Horizon Agentic RL
Mengsong Wu, Tianchen Zhou, Zhaochen Su, Zhenhua Liu, Zhe Cui, Aimin Zhou
AttnSR turns an LLM agent’s attention into step-level feedback to help it learn long-horizon tasks with fewer environment interactions.
MAC-v2: Long-horizon model-based value expansion from pixels
Kwanyoung Park, Sergey Levine
MAC-v2 helps offline RL agents imagine further into the future from pixels by learning useful latent dynamics and predicting actions in chunks.
On the Origin and Evolution of Training-Inference Mismatch in LLM Reinforcement Learning
Yu Li, Yufa Zhou, Tian Lan
The paper traces why LLM reinforcement learning can drift when sampling and training differ, then proposes a correction that detects when stale samples need resynchronization.