Learning Loco-Manipulation From SMPC Demonstrations With Sparse Offline-to-Online RL

This paper presents an offline-to-online reinforcement-learning pipeline for whole-body loco-manipulation that replaces manual dense-reward engineering with Sample-based Model Predictive Control, or SMPC, demonstrations generated entirely in simulation. Using MuJoCo Warp and a modified FastTD3 algorithm, the system initially mixes 50% expert transitions into the replay buffer, then phases them out after the policy begins succeeding, while optimizing only sparse task rewards. A frozen ReLIC whole-body stabilization controller converts high-level velocity, arm, torso-height, and pitch commands into dynamically stable joint targets, enabling neural policies to improve beyond their SMPC teacher rather than merely imitate it. GPU-vectorized SMPC produces 1M samples per hour, allowing the most difficult task to bootstrap from 4M samples collected within 4 GPU hours. The method reaches near-perfect simulated success and transfers five tasks—including reaching, box pushing, tire uprighting, tire rolling, and humanoid box pushing—to hardware on an arm-equipped Boston Dynamics Spot and a Unitree G1. Learned policies complete some tasks more than 50% faster than SMPC and reduce task-duration variability by 11–45%. Ablations show that larger datasets are needed for more complex coordination, while multimodal demonstrations can prevent learning; filtering SMPC data to a single behavioral mode is therefore critical.