NTH

JVT: Just Video Transformer makes pixel-space video diffusion simple

AuthorsJianjun Zhou, Hua Geng, Haoyu Guo

AffiliationsZhejiang University · Shanghai Innovation Institute · Shanghai AI Laboratoryhttps://zhoutimemachine.github.io/JVT

November 17, 2026 2 min read
Watch on YouTube
The one-line take

JVT shows that a straightforward Transformer can generate video directly in pixels—and even produce matching depth-like geometry alongside it.

Key results

136.5
UCF-101 FVD, JVT

Class-conditioned generation; Latte scores 477.97.

129.2
UCF-101 FVD, JVT with register guidance

Class-conditioned generation with register guidance.

79.5
RealEstate10K RTA@5

Joint RGB-disparity generation, compared with 72.4 for RGB-only JVT.

88.6
RealEstate10K AUC@30

Joint RGB-disparity generation, compared with 84.1 for RGB-only JVT.

What the paper found

JVT shows that video diffusion can operate directly on RGB pixels without a pretrained autoencoder, cascaded stages, detailers, or U-Net skip connections. It uses a DiT-style Transformer to predict clean video under a rectified-flow objective; its preferred design uses patch-first embeddings with a bottleneck width of 32, standard adaptive layer normalization, and optional register guidance. On UCF-101, JVT reaches an FVD of 136.5, compared with 477.97 for the latent-diffusion baseline Latte, and register guidance lowers the score to 129.2. The same pixel-space setup enables joint generation of camera-conditioned RGB and disparity, from which colored point clouds can be formed without an extra reconstruction model at inference. On RealEstate10K, adding disparity changes FVD from 27.0 to 27.5 while improving RTA@5 from 72.4 to 79.5 and AUC@30 from 84.1 to 88.6. The experiments also cover FaceForensics, SkyTimelapse, and Taichi-HD; camera-generation depth targets use Pi3X during training, and efficiency ablations run on an NVIDIA 4090. The results position a relatively simple Transformer as a practical pixel-space alternative to latent video diffusion, with aligned geometry as a native output rather than a separate prediction task.

Original abstract

Direct diffusion modeling of high-dimensional raw video data is challenging, and existing pixel-space methods often rely on specialized architectures. We introduce JVT (Just Video Transformer), a simple large-patch DiT-style Transformer for pixel-space video diffusion, without a pretrained autoencoder, auxiliary detailers, cascaded stages, or U-Net-style skip connections. We systematically examine video patch embeddings, register tokens, and adaptive layer normalization to identify design choices that balance generation quality and computational efficiency. The resulting JVT design achieves generation quality competitive with latent diffusion baselines in both class-conditioned and unconditional settings. The pixel-space formulation of JVT also naturally supports joint generation of pixel-aligned modalities. We demonstrate this capability through camera-conditioned RGB-disparity generation, which improves camera following relative to RGB-only baselines and yields explicit scene geometry without an additional reconstruction model at inference. These results highlight the potential of simple DiT-style Transformers for pixel-space video diffusion, and we hope our findings offer useful insights and practical guidance for future research.

Read the original paper

More in Diffusion Models

Browse all 82 papers →