NTH

ExRL: Sample-Efficient Online Adaptation of Long-Chunk Policies via Execution Length Control

AuthorsXu Luo, Jiaying Yang, Hao Wu, Junlin Xie, Zijin Hong, Ji Zhang, Lianli Gao, Jingkuan Song

AffiliationsUESTC Tongji University The Hong Kong Polytechnic University https://exrl-project.github.io Learning efficiency on Tool Hang Stack cube Power insertion 1.0 1.0 1.0 success-rate AUC Chunk size

November 17, 2026 2 min read
Watch on YouTube
The one-line take

ExRL helps robots adapt long action plans using fewer interactions by learning when to switch to the next chunk.

Key results

50
Real-world action chunk size

The π0.5 policies predict chunks of 50 actions.

90%
LIBERO π0.5 success

ExRL exceeds 90% success on LIBERO.

200000
LIBERO environment steps

Steps to exceed 90% success on LIBERO.

17
Stacking and folding successful trials

17 of 20 trials succeeded for each of cube stacking and clothing folding.

22
Power insertion successful trials

22 of 25 real-world trials succeeded.

104
Parts organization completed stages

104 of 125 real-world stages were completed.

What the paper found

ExRL adapts long-chunk robot policies by learning how many actions of each predicted chunk to execute before replanning, rather than optimizing the high-dimensional actions themselves. Its lightweight adapter uses off-policy ExRL-Q or discrete ExRL-SAC, with a duration-aware semi-Markov TD target and conservative Q-learning; optional behavior cloning can transfer successful online behavior back into the base policy. The method adapts Diffusion Policy, ACT, and generalist models including π0.5, π0-FAST, and Cosmos Policy, using chunk size H=50 in its real-world trials. On LIBERO, ExRL raises π0.5 from nearly 0% to over 90% success within 200000 environment steps. On a real robot, it achieves 17 successful trials out of 20 for both cube stacking and clothing folding, 22 out of 25 for power insertion, and completes 104 of 125 parts-organization stages. These results show that state- and chunk-conditioned execution-length control can improve diverse frozen policies with limited interaction, while replanning more often near contact and allowing longer executions when predictions remain useful.

Original abstract

Predicting long action chunks, often spanning dozens of timesteps, has become a key ingredient in increasingly capable robot policies. In this paper, we find that adapting these long-chunk policies with existing reinforcement learning (RL) methods can require many online samples due to the high dimensionality of the action chunks being optimized, making such adaptation impractical in real-world scenarios where interaction data are costly to collect. Our key insight is to shift adaptation from optimizing actions to optimizing how long to execute each predicted action chunk. This keeps the adaptation space low-dimensional while its expressivity grows with the chunk size. We thus introduce Execution at the Right Length (ExRL), a simple approach which augments the base policy with a lightweight adapter that predicts the execution length of each chunk and is trained using off-policy RL algorithms. Across simulation benchmarks and real-world robotic tasks, ExRL enables sample-efficient adaptation of a diverse range of long-chunk policies with minimal task-specific hyperparameter tuning.

Read the original paper

More in Robotics

Browse all 68 papers →