NTH

Training Stability under Adaptive Curricula

AuthorsJohan Obando-Ceron, Walter Mayor, Michael Dennis, Aaron Courville, Pablo Samuel Castro

AffiliationsMila – Québec AI Institute · Université de Montréal · Google DeepMind · CIFAR AI Chair

November 16, 2026 2 min read
Watch on YouTube
The one-line take

Adaptive training environments can undermine an RL agent’s ability to learn, so stable learners are essential for lasting performance.

Key results

2B
Standard interaction budget

Environment interactions used in standard experiments.

5
Random seeds

Seeds used for standard experiments.

5B
Extended training horizon

Interactions in the long-horizon PLR experiment.

128
Limited rollout parallelism

Parallel environments where stabilization still improved performance.

2
Weight clipping factor

Clipping range scale used in the selected stabilization setup.

51
HL-Gauss value bins

Categorical value support resolution in the selected stabilization setup.

What the paper found

A study involving Google DeepMind finds that adaptive reinforcement-learning curricula can keep selecting challenging environments even as the agent loses the ability to learn from them. In Kinetix, PPO agents trained with Prioritized Level Replay showed parameter growth, weaker ranking of outcomes by predicted value, and less diverse representations, especially on complex tasks. The proposed stabilization combines weight clipping with HL-Gauss, which trains value predictions as categorical distributions. Using a clipping factor of 2 and 51 value bins, the combination improved learning without changing curriculum rules or interaction budgets, and also helped with domain randomization, SFL, and ACCEL. The standard experiments used 2B environment interactions and 5 seeds; with training extended to 5B interactions, the stabilized agent continued improving while the baseline degraded. Stabilization also helped under limited rollout parallelism, including 128 environments, and supported zero-shot transfer and retention after task-distribution shifts. The central point is that curriculum quality depends not only on which tasks are selected, but also on whether the learner remains trainable on them.

Original abstract

Unsupervised Environment Design (UED) approaches train reinforcement learning (RL) agents on diverse curricula that may evolve with their capabilities. While prior work has largely focused on how these curricula are constructed, we study whether their students remain trainable under the resulting distributions. Across multiple UED mechanisms, environment complexities, and extended training horizons, we identify recurring learner-side pathologies, including parameter growth, degraded value--outcome ordering, and loss of representational diversity, that accompany stagnation and performance degradation. These findings connect limitations in UED to optimization failures previously identified in conventional deep RL, while showing how adaptive curricula couple the learner's optimization state to its future training distribution. Controlled interventions show that mitigating selected pathologies improves learning. Stabilizing the learner improves long-horizon performance and zero-shot generalization without changing the curriculum rule or increasing environment interaction. Our results establish learner trainability as a complementary design principle for UED. Effective curricula require students that remain capable of learning from the distributions they produce.

Read the original paper

More in Reinforcement Learning

Browse all 69 papers →
01Rl

AttnSR: Attention Step Reward for Long-Horizon Agentic RL

Mengsong Wu, Tianchen Zhou, Zhaochen Su, Zhenhua Liu, Zhe Cui, Aimin Zhou

AttnSR turns an LLM agent’s attention into step-level feedback to help it learn long-horizon tasks with fewer environment interactions.

Read analysis