NTH

When Does Latent Prediction Error Support Physical Surprise Claims in Video World Models?

AuthorsAnton Hristov, Aleksandar Slavov, Vasil Vasilev

AffiliationsTechnology School Electronic Systems (TUES) · Baba Tonka High School of Mathematics

November 9, 2026 2 min read
Watch on YouTube
The one-line take

Video world models can encode physical information, but their prediction errors may be driven by appearance or other factors rather than physical surprise.

Key results

57.9%
V-JEPA2-G IntPhys2 physics accuracy

Frozen-feature classification of possible versus impossible events.

8.7%
Advantage over VideoMAE2-G

Percentage-point gain in IntPhys2 physics classification accuracy.

97.7%
IntPhys2 appearance accuracy

V-JEPA2-G frozen-feature classification of appearance labels.

256
Kubric configurations

Controlled crossed-support configurations testing error versus physical validity.

53.9%
Fixed-threshold balanced accuracy

Kubric clean evaluation clips under a threshold fitted on separate calibration scenes.

68.95%
False alarms on lawful clips

Kubric clean evaluation clips; lawful events incorrectly flagged as impossible.

What the paper found

This study tests whether a video world model’s prediction error can be trusted as a measure of physical surprise. On IntPhys2, frozen V-JEPA2-G features classify possible versus impossible events with 57.9% accuracy, an 8.7-percentage-point advantage over VideoMAE2-G, while appearance labels reach 97.7% accuracy—showing that physical information is present but appearance is readily decoded too. In deeper V-JEPA2-G layers, hue shifts shrink from 57% to 8% of the distance between unrelated videos, yet this geometric change adds little ability to predict the model’s own error. The decisive test uses Kubric to create 256 configurations where the same motion is lawful with support and impossible over a gap. Error follows the motion rather than physical validity, reversing its impossible-versus-possible ordering across settings. A fixed threshold achieves only 53.9% balanced accuracy and falsely flags 68.95% of lawful clips. Other controls reinforce the caution: repeating a frame can raise error, persistent recoloring can lower it, and LeWorldModel, DINO-WM, and JEPA-WM respond both to incorrect actions and to appearance changes. The central finding is that representation-level physical information and action sensitivity do not make native prediction error a reliable, transferable measure of physical surprise.

Original abstract

Prediction errors from video world models are often treated as physical surprise, but they can also reflect appearance, static content, motion, or ordinary prediction difficulty. We present an evaluation framework that separates what frozen representations contain, how they organize appearance and physical differences, and what changes a model's own prediction error. On IntPhys2, a benchmark of physically possible and impossible events, we compare representations from latent prediction, masked pixel reconstruction, and simple visual statistics. Appearance remains easy to decode, while V-JEPA2 exposes modest information about whether events are possible or impossible. Looking at the latent geometry, appearance changes become smaller compared with differences between videos, their directions line up more consistently across scenes, and their overlap with physical changes decreases in deeper layers. Across action-conditioned models, incorrect actions raise prediction error, while appearance changes also produce substantial responses. In Physion++ video clips, repeating a single frame can produce higher error than the original clip, while changing color throughout the clip can lower it. Using Kubric, a simulation and rendering framework, we find that impossible events can produce lower error than possible events. A fixed error threshold also incorrectly flags many possible events as impossible. A model can therefore represent physical information and respond to incorrect actions without its native prediction error reliably measuring physical surprise.

Read the original paper

More in World Models

Browse all 59 papers →
01World Model

Duo: A Reciprocal World Model for Full-Duplex Interaction

Hengyuan Cao, Biao Gong, Guoshan Lu, Qiyuan Zhang, Lin Liu, Yiding Tian, Yilei Jiang, Yuxin Ma, Chengzhang Wu, Kun Chen, Yiqian Xie, Ting Liu, Xin Zheng, Xijun Gu, Tianren Ma, Jiancheng Pan, Chi Zhang, Xuzhe Zheng, Siyi Li, Zhenzhong Lan

Duo aims to create an always-on audiovisual assistant that can perceive, reason, and respond continuously, including during interruptions.

Read analysis
02World Model

PanDWM: Panoramic 4D Driving World Models

Jiajun Sun, Zeyu Zhang, Zeyu Ren, Guangfeng Jiang, Yang Zhao, Hao Tang

PanDWM generates a continuous 360-degree driving view and reconstructs it into a fast 4D scene.

Read analysis
03World Model

WorldRuler: Interactive World Modeling under Configurable Rules

Mingliang Zhai, Chuanhao Li, Jiaming Tan, Zhen Li, Yuwei Wu, Yunde Jia, Zian Meng, Yuanyang Yin, Kaipeng Zhang

WorldRuler aims to make generated interactive worlds obey customizable rules, even after saving and restoring the world.

Read analysis