Beyond Pixels: From Video Priors to 4D Worlds

Beyond Pixels proposes Latent-to-4D, a method that converts a video generator’s final denoised VAE latent directly into dynamic geometry, cameras, and motion, bypassing the usual RGB decoding and reconstruction cascade. Its Latent-to-4D Alignment and Refinement, or L4AR, first matches the latent grid with a learned 3D convolution, then combines frame-wise and global spatiotemporal attention before a pretrained 4D decoder predicts world-space point maps and camera trajectories. Training uses roughly 1,143 annotated reconstruction clips while freezing the video generators, VAE, and main Transformer weights, allowing one checkpoint to operate across 3 compatible DiTs that share the Wan VAE, including Wan2.1 text-to-video and Wan2.2 image-to-video models. On the locked Text4D-200 and I4D-200 benchmarks, direct latent lifting improves projection-based DINO-F1 over matched Wan-plus-4RC baselines by 2.88–3.45 points for text conditioning and 5.81 points for image conditioning; CogVideoX-5B remains stronger on some RGB-reference metrics, so the gains are concentrated in geometric consistency rather than every measure. Human evaluations favor the method for geometry, completeness, temporal stability, and overall quality, with text-to-4D geometry preference reaching 59.2% and image-to-4D completeness reaching 72.1%. The central result is a reusable latent interface: compatible video models retain their original text, image, motion, pose, trajectory, and navigation controls while sharing a single geometry-supervised 4D pathway, though the evidence is limited to a common-VAE setting and projection metrics do not establish metric accuracy.