NTH

MAC-v2: Long-horizon model-based value expansion from pixels

AuthorsKwanyoung Park, Sergey Levine

AffiliationsUC Berkeley · https://kwanyoungpark.github.io/MAC-v2/

November 16, 2026 2 min read
Watch on YouTube
The one-line take

MAC-v2 helps offline RL agents imagine further into the future from pixels by learning useful latent dynamics and predicting actions in chunks.

Key results

100
Action-chunked rollout horizon

Environment steps imagined using 10 recursive predictions with 10-step action chunks.

25
Visual OGBench tasks

Visual offline reinforcement-learning tasks used for evaluation.

95%
MAC-v2 visual-puzzle-3x3 score

OGBench success score, compared with 22% for the best prior method.

22%
Best prior visual-puzzle-3x3 score

OGBench score on visual-puzzle-3x3.

66
MMBench2 pygame score

Normalized score, versus 58 for the strongest baseline.

80%
LIBERO-10 success rate

Matches the behavior-cloning baseline.

What the paper found

MAC-v2 addresses a central challenge in offline reinforcement learning from images: imagined trajectories can become unreliable when errors in visual representations and learned dynamics compound over time. It trains a reconstruction-free latent world model using future-state prediction, reward prediction, and behavior cloning with flow matching, so the representation focuses on information useful for control rather than reproducing every pixel. It also predicts action chunks of 10 steps at a time, allowing a 100-environment-step imagined rollout with only 10 recursive dynamics predictions. A Q-function ranks behavior-policy action candidates, and values are learned from the resulting on-policy model rollouts. Across 25 visual OGBench tasks, MAC-v2 scores 95% on visual-puzzle-3x3, compared with 22% for the best prior method. On MMBench2, it achieves a score of 66 on the pygame domain, versus 58 for the strongest baseline. On LIBERO-10, it reaches 80%, matching behavior cloning. Ablations show that longer rollouts, action chunking, and training the latent encoder with both world-model and control objectives are important for performance.

Original abstract

We study whether \emph{long-horizon model-based value expansion} can serve as an effective framework in pixel-based offline reinforcement learning (RL). Long-horizon model-based value expansion from pixels is substantially harder than state-based environments, as the agent must learn both the representation and its dynamics, and errors from either can accumulate recursively over imagined trajectories. We introduce MAC-v2, which enables long-horizon model-based expansion from pixels by training a reconstruction-free latent world model with action chunking. By learning representations from prediction and control objectives, MAC-v2 does not need to encode control-irrelevant visual details that can hinder the dynamics prediction. Action chunking further mitigates the prediction error by reducing the number of recursive model predictions required for value expansion. Across diverse visual benchmarks including visual OGBench, MMBench2, Robomimic, and LIBERO-10, MAC-v2 achieves strong performance against model-free and model-based offline RL baselines. Our analyses show that long-horizon model-based value expansion is critical for the performance of MAC-v2, and that both action chunking and latent learning objectives are essential for reliable long-horizon latent imagination.

Read the original paper

More in Reinforcement Learning

Browse all 69 papers →
01Rl

AttnSR: Attention Step Reward for Long-Horizon Agentic RL

Mengsong Wu, Tianchen Zhou, Zhaochen Su, Zhenhua Liu, Zhe Cui, Aimin Zhou

AttnSR turns an LLM agent’s attention into step-level feedback to help it learn long-horizon tasks with fewer environment interactions.

Read analysis
02Rl

Training Stability under Adaptive Curricula

Johan Obando-Ceron, Walter Mayor, Michael Dennis, Aaron Courville, Pablo Samuel Castro

Adaptive training environments can undermine an RL agent’s ability to learn, so stable learners are essential for lasting performance.

Read analysis