Training Controllable World Models via Action Influence Maximization

Action-conditioned world models can ignore commands when expert demonstrations contain too little variation in actions: the model can predict expert-like outcomes without learning how alternative actions change the future. This paper introduces Action Influence Maximization, or AIM, which treats conditional mutual information between actions and next states as an upper bound on the controllability a model can learn from its data. AIM estimates a reward for action-informative transitions and underrepresented states, then uses conservative offline reinforcement learning to collect targeted experience. Its implementation combines flow-matching density models with an Implicit Q-Learning critic and Advantage-Weighted Regression; the evaluated robot world model is a DreamerV4-like latent Diffusion Forcing model. Across Four Rooms and pixel-based Push-T manipulation, action influence correlates with simulation fidelity at 0.979, and AIM is the only tested method that consistently improves Push-T end-effector controllability. Adding 500 AIM episodes to 3K demonstrations matches the end-effector controllability of training on 300K expert episodes—an 85× data-efficiency gain—and improves BC joint recall by 69% over the PPO-3K baseline. The results suggest that adding data helps most when it adds missing action consequences, not merely more transitions; the experiments are in simulation, and AIM’s density-reward retraining is computationally demanding. The acknowledgments cite support from Microsoft Research, NVIDIA, and the Gemini Academic Program, rather than contributions to the method.