JVT: Just Video Transformer makes pixel-space video diffusion simple
AuthorsJianjun Zhou, Hua Geng, Haoyu Guo
AffiliationsZhejiang University · Shanghai Innovation Institute · Shanghai AI Laboratoryhttps://zhoutimemachine.github.io/JVT
JVT shows that a straightforward Transformer can generate video directly in pixels—and even produce matching depth-like geometry alongside it.
Key results
Class-conditioned generation; Latte scores 477.97.
Class-conditioned generation with register guidance.
Joint RGB-disparity generation, compared with 72.4 for RGB-only JVT.
Joint RGB-disparity generation, compared with 84.1 for RGB-only JVT.
What the paper found
JVT shows that video diffusion can operate directly on RGB pixels without a pretrained autoencoder, cascaded stages, detailers, or U-Net skip connections. It uses a DiT-style Transformer to predict clean video under a rectified-flow objective; its preferred design uses patch-first embeddings with a bottleneck width of 32, standard adaptive layer normalization, and optional register guidance. On UCF-101, JVT reaches an FVD of 136.5, compared with 477.97 for the latent-diffusion baseline Latte, and register guidance lowers the score to 129.2. The same pixel-space setup enables joint generation of camera-conditioned RGB and disparity, from which colored point clouds can be formed without an extra reconstruction model at inference. On RealEstate10K, adding disparity changes FVD from 27.0 to 27.5 while improving RTA@5 from 72.4 to 79.5 and AUC@30 from 84.1 to 88.6. The experiments also cover FaceForensics, SkyTimelapse, and Taichi-HD; camera-generation depth targets use Pi3X during training, and efficiency ablations run on an NVIDIA 4090. The results position a relatively simple Transformer as a practical pixel-space alternative to latent video diffusion, with aligned geometry as a native output rather than a separate prediction task.
Original abstract
Direct diffusion modeling of high-dimensional raw video data is challenging, and existing pixel-space methods often rely on specialized architectures. We introduce JVT (Just Video Transformer), a simple large-patch DiT-style Transformer for pixel-space video diffusion, without a pretrained autoencoder, auxiliary detailers, cascaded stages, or U-Net-style skip connections. We systematically examine video patch embeddings, register tokens, and adaptive layer normalization to identify design choices that balance generation quality and computational efficiency. The resulting JVT design achieves generation quality competitive with latent diffusion baselines in both class-conditioned and unconditional settings. The pixel-space formulation of JVT also naturally supports joint generation of pixel-aligned modalities. We demonstrate this capability through camera-conditioned RGB-disparity generation, which improves camera following relative to RGB-only baselines and yields explicit scene geometry without an additional reconstruction model at inference. These results highlight the potential of simple DiT-style Transformers for pixel-space video diffusion, and we hope our findings offer useful insights and practical guidance for future research.
Read the original paperMore in Diffusion Models
Browse all 82 papers →FewStepRL: Fine-tuning Few-step Diffusion Models with Reinforcement Learning
Haitao Lin, Minsi Ren, Yang Li, Xingtong Ge, Zheyuan Hu, Peiyan Hu, Tailin Wu
FewStepRL adapts multiple reinforcement-learning fine-tuning methods to make reward-driven improvements to fast, few-step diffusion models.
Restoring Diversity in Distilled Image and Video Generators by Probing Activations
Furkan Horoz, Ozgur Kara, Kaan Oktay, Adil Kaan Akan, Atakan Kara, Anand Bhattad, James Matthew Rehg
The paper restores variety in fast distilled image and video generators by finding and correcting a collapse in their internal activation directions.
BYOD: Build Your Own DLM - Efficient Autoregressive-to-Diffusion Language Model Conversion
Ruurd J. A. Kuiper, Robert A. Bagheri
BYOD shows how to turn existing language models into diffusion generators with small LoRA adapters, while highlighting the quality gap that remains.