PanDWM: Panoramic 4D Driving World Models
AuthorsJiajun Sun, Zeyu Zhang, Zeyu Ren, Guangfeng Jiang, Yang Zhao, Hao Tang
AffiliationsThe University of Melbourne · Peking University · La Trobe University
PanDWM generates a continuous 360-degree driving view and reconstructs it into a fast 4D scene.
Key results
Steps used to generate each panoramic window.
Panoramic-band image quality score on nuScenes Val-150.
Video score for the first 16-frame window on nuScenes Val-150.
Video score for the second 16-frame window on nuScenes Val-150.
Seconds per frame; reported as 8–46 times faster than multi-camera baselines.
Static-region depth error for PanDWM’s 4D Gaussian scenes.
What the paper found
PanDWM replaces six separately generated driving-camera videos with one 360° panoramic band, turning cross-camera alignment into image continuity. It splits the band into two overlapping halves and uses a causal diffusion student, distilled from a Stable Diffusion 3 teacher and trained on its own rollouts, to generate 16 future frames per window in four denoising steps. A Stable Diffusion 3.5 VAE encodes the video, while layout conditions and calibrated camera rays guide generation. For 3D output, a feed-forward decoder places Gaussians along each pixel’s source-camera ray, using Depth Anything 3 to refine their ranges; static Gaussians from neighboring frames then complete the scene for 4D novel-view rendering. On nuScenes Val-150, PanDWM reports FID 10.1, FVD-1 37.4, and FVD-2 62.1, while running at 1.96 seconds per frame—8–46 times faster than the multi-camera baselines. Its 4D reconstruction reaches a static-depth AbsRel of 0.316. The paper also discloses using Claude from Anthropic and Codex from OpenAI for implementation and manuscript assistance.
Original abstract
Driving world models usually generate videos from six pinhole cameras with separate optical centres and small overlaps, and must learn to keep these views consistent. We present PanDWM, which stitches them into one $360^\circ$ panoramic band. Camera boundaries then fall inside one image, so keeping the views consistent means keeping the image continuous, and each pixel still records its camera ray. Conditioned on layouts, a causal student generates the band window by window in four steps; it is distilled from a causal teacher and trained on its own rollouts. A feed-forward decoder lifts each frame along these rays into 3D Gaussians, and static Gaussians from neighbouring frames complete them into 4D scenes. On nuScenes, scored in one stitched $360^\circ$ view, PanDWM has lower FID and FVD than the multi-camera baselines at $8$-$46\times$ their speed, and keeps its 50-step teacher's FID while lowering FVD by 22-25%.
Read the original paperMore in World Models
Browse all 59 papers →Duo: A Reciprocal World Model for Full-Duplex Interaction
Hengyuan Cao, Biao Gong, Guoshan Lu, Qiyuan Zhang, Lin Liu, Yiding Tian, Yilei Jiang, Yuxin Ma, Chengzhang Wu, Kun Chen, Yiqian Xie, Ting Liu, Xin Zheng, Xijun Gu, Tianren Ma, Jiancheng Pan, Chi Zhang, Xuzhe Zheng, Siyi Li, Zhenzhong Lan
Duo aims to create an always-on audiovisual assistant that can perceive, reason, and respond continuously, including during interruptions.
WorldRuler: Interactive World Modeling under Configurable Rules
Mingliang Zhai, Chuanhao Li, Jiaming Tan, Zhen Li, Yuwei Wu, Yunde Jia, Zian Meng, Yuanyang Yin, Kaipeng Zhang
WorldRuler aims to make generated interactive worlds obey customizable rules, even after saving and restoring the world.
Future Representation Distillation for Imagination-free World Action Models
Jaehyeong Jo, Sangwon Jang, Sihyun Yu, Seonghyeon Ye, Yekyung Nah, Hojung Jung, Kimin Lee, Sung Ju Hwang
DreamFree teaches a robot to use a video model’s learned sense of the future without having to generate that future frame by frame.