ExRL: Sample-Efficient Online Adaptation of Long-Chunk Policies via Execution Length Control
AuthorsXu Luo, Jiaying Yang, Hao Wu, Junlin Xie, Zijin Hong, Ji Zhang, Lianli Gao, Jingkuan Song
AffiliationsUESTC Tongji University The Hong Kong Polytechnic University https://exrl-project.github.io Learning efficiency on Tool Hang Stack cube Power insertion 1.0 1.0 1.0 success-rate AUC Chunk size
ExRL helps robots adapt long action plans using fewer interactions by learning when to switch to the next chunk.
Key results
The π0.5 policies predict chunks of 50 actions.
ExRL exceeds 90% success on LIBERO.
Steps to exceed 90% success on LIBERO.
17 of 20 trials succeeded for each of cube stacking and clothing folding.
22 of 25 real-world trials succeeded.
104 of 125 real-world stages were completed.
What the paper found
ExRL adapts long-chunk robot policies by learning how many actions of each predicted chunk to execute before replanning, rather than optimizing the high-dimensional actions themselves. Its lightweight adapter uses off-policy ExRL-Q or discrete ExRL-SAC, with a duration-aware semi-Markov TD target and conservative Q-learning; optional behavior cloning can transfer successful online behavior back into the base policy. The method adapts Diffusion Policy, ACT, and generalist models including π0.5, π0-FAST, and Cosmos Policy, using chunk size H=50 in its real-world trials. On LIBERO, ExRL raises π0.5 from nearly 0% to over 90% success within 200000 environment steps. On a real robot, it achieves 17 successful trials out of 20 for both cube stacking and clothing folding, 22 out of 25 for power insertion, and completes 104 of 125 parts-organization stages. These results show that state- and chunk-conditioned execution-length control can improve diverse frozen policies with limited interaction, while replanning more often near contact and allowing longer executions when predictions remain useful.
Original abstract
Predicting long action chunks, often spanning dozens of timesteps, has become a key ingredient in increasingly capable robot policies. In this paper, we find that adapting these long-chunk policies with existing reinforcement learning (RL) methods can require many online samples due to the high dimensionality of the action chunks being optimized, making such adaptation impractical in real-world scenarios where interaction data are costly to collect. Our key insight is to shift adaptation from optimizing actions to optimizing how long to execute each predicted action chunk. This keeps the adaptation space low-dimensional while its expressivity grows with the chunk size. We thus introduce Execution at the Right Length (ExRL), a simple approach which augments the base policy with a lightweight adapter that predicts the execution length of each chunk and is trained using off-policy RL algorithms. Across simulation benchmarks and real-world robotic tasks, ExRL enables sample-efficient adaptation of a diverse range of long-chunk policies with minimal task-specific hyperparameter tuning.
Read the original paperMore in Robotics
Browse all 68 papers →ELA: End-to-End Latent Action Policy Learning from Video Demonstrations
Cheng-Chun Hsu, Peter Stone
ELA turns plentiful robot videos into useful control experience by jointly learning latent actions, policies, and action decoders.
GRASP: Gradient-Guided Sampling for Contact-Rich Physics-Based Motion Retargeting
Yang Liu, Tong Wang, Yiming Bao, Heng Yang
GRASP uses past simulation rollouts to guide robot motion retargeting, producing contact-rich movements that work in simulation and on real hardware.
PERA: Plug-in Experts for Robotic Actions in Continual Whole-Body Control
Shangchao Hu, Yadan Sun, Xibo Yu, Wei Huang
PERA teaches humanoid robots new skills by plugging small experts into an existing controller without disrupting its original motions.