NTH

PanDWM: Panoramic 4D Driving World Models

AuthorsJiajun Sun, Zeyu Zhang, Zeyu Ren, Guangfeng Jiang, Yang Zhao, Hao Tang

AffiliationsThe University of Melbourne · Peking University · La Trobe University

November 20, 2026 2 min read
Watch on YouTube
The one-line take

PanDWM generates a continuous 360-degree driving view and reconstructs it into a fast 4D scene.

Key results

4
Student denoising steps

Steps used to generate each panoramic window.

10.1
Val-150 FID

Panoramic-band image quality score on nuScenes Val-150.

37.4
Val-150 FVD-1

Video score for the first 16-frame window on nuScenes Val-150.

62.1
Val-150 FVD-2

Video score for the second 16-frame window on nuScenes Val-150.

1.96
Runtime per frame

Seconds per frame; reported as 8–46 times faster than multi-camera baselines.

0.316
4D static-depth AbsRel

Static-region depth error for PanDWM’s 4D Gaussian scenes.

What the paper found

PanDWM replaces six separately generated driving-camera videos with one 360° panoramic band, turning cross-camera alignment into image continuity. It splits the band into two overlapping halves and uses a causal diffusion student, distilled from a Stable Diffusion 3 teacher and trained on its own rollouts, to generate 16 future frames per window in four denoising steps. A Stable Diffusion 3.5 VAE encodes the video, while layout conditions and calibrated camera rays guide generation. For 3D output, a feed-forward decoder places Gaussians along each pixel’s source-camera ray, using Depth Anything 3 to refine their ranges; static Gaussians from neighboring frames then complete the scene for 4D novel-view rendering. On nuScenes Val-150, PanDWM reports FID 10.1, FVD-1 37.4, and FVD-2 62.1, while running at 1.96 seconds per frame—8–46 times faster than the multi-camera baselines. Its 4D reconstruction reaches a static-depth AbsRel of 0.316. The paper also discloses using Claude from Anthropic and Codex from OpenAI for implementation and manuscript assistance.

Original abstract

Driving world models usually generate videos from six pinhole cameras with separate optical centres and small overlaps, and must learn to keep these views consistent. We present PanDWM, which stitches them into one $360^\circ$ panoramic band. Camera boundaries then fall inside one image, so keeping the views consistent means keeping the image continuous, and each pixel still records its camera ray. Conditioned on layouts, a causal student generates the band window by window in four steps; it is distilled from a causal teacher and trained on its own rollouts. A feed-forward decoder lifts each frame along these rays into 3D Gaussians, and static Gaussians from neighbouring frames complete them into 4D scenes. On nuScenes, scored in one stitched $360^\circ$ view, PanDWM has lower FID and FVD than the multi-camera baselines at $8$-$46\times$ their speed, and keeps its 50-step teacher's FID while lowering FVD by 22-25%.

Read the original paper

More in World Models

Browse all 59 papers →
01World Model

Duo: A Reciprocal World Model for Full-Duplex Interaction

Hengyuan Cao, Biao Gong, Guoshan Lu, Qiyuan Zhang, Lin Liu, Yiding Tian, Yilei Jiang, Yuxin Ma, Chengzhang Wu, Kun Chen, Yiqian Xie, Ting Liu, Xin Zheng, Xijun Gu, Tianren Ma, Jiancheng Pan, Chi Zhang, Xuzhe Zheng, Siyi Li, Zhenzhong Lan

Duo aims to create an always-on audiovisual assistant that can perceive, reason, and respond continuously, including during interruptions.

Read analysis
02World Model

WorldRuler: Interactive World Modeling under Configurable Rules

Mingliang Zhai, Chuanhao Li, Jiaming Tan, Zhen Li, Yuwei Wu, Yunde Jia, Zian Meng, Yuanyang Yin, Kaipeng Zhang

WorldRuler aims to make generated interactive worlds obey customizable rules, even after saving and restoring the world.

Read analysis