SmartDirector: Keyframe-Conditioned Cinematic Video Generation with Narrative Pacing Control

SmartDirector is a two-stage cinematic video generation framework for arbitrary keyframe conditioning that targets single-shot synthesis, multi-shot narrative generation, and video extension. Built on an internal 32B diffusion backbone for Director-Gen and Wan-2.2-5B for Director-SR, it replaces direct latent keyframe insertion with a Multi-Chunk VAE that encodes each keyframe as the first frame of its own chunk, then applies full spatio-temporal attention in a DiT and a custom MC-RoPE temporal scheme to avoid boundary artifacts. The second stage uses high-resolution keyframes as semantic anchors to super-resolve low-resolution outputs, recovering faces, text, and other fine details. Trained on curated cinematic footage from copyright-free movies plus UltraVideo for super-resolution, SmartDirector is evaluated on a benchmark of 250 single-shot and 250 multi-shot videos at 24 FPS and native 1080p or higher. It sharply outperforms Dreamina Multiframes, cutting FVD from 226.85 to 41.12 in single-shot and from 251.83 to 65.65 in multi-shot, while raising Gemini-3-Pro semantic scores from 83.87 to 91.30 and from 59.32 to 88.48, with narrative coherence gains especially large. In human study, 30 participants judged 500 video pairs, and SmartDirector achieved a 54.73% win rate in multi-shot overall quality. As a standalone super-resolution method, it also reduces LPIPS versus SparkVSR on all four benchmarks, showing that keyframe-conditioned generation and restoration can be unified in one narrative-aware pipeline; related model and evaluation infrastructure comes from Google Gemini, Qwen3-VL, and NVIDIA training on 40 GPUs for 20,000 steps.