MAC-v2: Long-horizon model-based value expansion from pixels
AuthorsKwanyoung Park, Sergey Levine
AffiliationsUC Berkeley · https://kwanyoungpark.github.io/MAC-v2/
MAC-v2 helps offline RL agents imagine further into the future from pixels by learning useful latent dynamics and predicting actions in chunks.
Key results
Environment steps imagined using 10 recursive predictions with 10-step action chunks.
Visual offline reinforcement-learning tasks used for evaluation.
OGBench success score, compared with 22% for the best prior method.
OGBench score on visual-puzzle-3x3.
Normalized score, versus 58 for the strongest baseline.
Matches the behavior-cloning baseline.
What the paper found
MAC-v2 addresses a central challenge in offline reinforcement learning from images: imagined trajectories can become unreliable when errors in visual representations and learned dynamics compound over time. It trains a reconstruction-free latent world model using future-state prediction, reward prediction, and behavior cloning with flow matching, so the representation focuses on information useful for control rather than reproducing every pixel. It also predicts action chunks of 10 steps at a time, allowing a 100-environment-step imagined rollout with only 10 recursive dynamics predictions. A Q-function ranks behavior-policy action candidates, and values are learned from the resulting on-policy model rollouts. Across 25 visual OGBench tasks, MAC-v2 scores 95% on visual-puzzle-3x3, compared with 22% for the best prior method. On MMBench2, it achieves a score of 66 on the pygame domain, versus 58 for the strongest baseline. On LIBERO-10, it reaches 80%, matching behavior cloning. Ablations show that longer rollouts, action chunking, and training the latent encoder with both world-model and control objectives are important for performance.
Original abstract
We study whether \emph{long-horizon model-based value expansion} can serve as an effective framework in pixel-based offline reinforcement learning (RL). Long-horizon model-based value expansion from pixels is substantially harder than state-based environments, as the agent must learn both the representation and its dynamics, and errors from either can accumulate recursively over imagined trajectories. We introduce MAC-v2, which enables long-horizon model-based expansion from pixels by training a reconstruction-free latent world model with action chunking. By learning representations from prediction and control objectives, MAC-v2 does not need to encode control-irrelevant visual details that can hinder the dynamics prediction. Action chunking further mitigates the prediction error by reducing the number of recursive model predictions required for value expansion. Across diverse visual benchmarks including visual OGBench, MMBench2, Robomimic, and LIBERO-10, MAC-v2 achieves strong performance against model-free and model-based offline RL baselines. Our analyses show that long-horizon model-based value expansion is critical for the performance of MAC-v2, and that both action chunking and latent learning objectives are essential for reliable long-horizon latent imagination.
Read the original paperMore in Reinforcement Learning
Browse all 69 papers →AttnSR: Attention Step Reward for Long-Horizon Agentic RL
Mengsong Wu, Tianchen Zhou, Zhaochen Su, Zhenhua Liu, Zhe Cui, Aimin Zhou
AttnSR turns an LLM agent’s attention into step-level feedback to help it learn long-horizon tasks with fewer environment interactions.
Training Stability under Adaptive Curricula
Johan Obando-Ceron, Walter Mayor, Michael Dennis, Aaron Courville, Pablo Samuel Castro
Adaptive training environments can undermine an RL agent’s ability to learn, so stable learners are essential for lasting performance.
On the Origin and Evolution of Training-Inference Mismatch in LLM Reinforcement Learning
Yu Li, Yufa Zhou, Tian Lan
The paper traces why LLM reinforcement learning can drift when sampling and training differ, then proposes a correction that detects when stale samples need resynchronization.