OmniDirector: General Multi-Shot Camera Cloning without Cross-Paired Data

OmniDirector, developed by the Kuaishou Technology Kling team with Tsinghua University and Peking University collaborators, tackles general multi-shot camera cloning by replacing brittle camera-parameter injection and scarce cross-paired data with a visual camera grid: reference videos are decomposed into camera poses, rendered as motion in an empty 3D scene, and fed into a multimodal diffusion transformer as a spatiotemporal conditioning stream. The method is trained on a million-scale camera grid–video dataset built from 1.8M internet videos, then fine-tuned with a self-reconstruction objective on 30% of samples to force the model to learn geometry rather than appearance leakage. At inference, a hierarchical Prompt Expansion agent uses Qwen3-VL to generate inter-shot and intra-shot camera descriptions and fuse them with the user prompt and reference image, while adaptive classifier-free guidance separates camera structure from semantic refinement. On a 1,094-sample evaluation set, OmniDirector outperforms CamCloneMaster, Seedance2.0, and LTX-LoRA, reaching 2.64° relative rotation error, 16.84° relative translation error, 83.18% rotation precision, 72.74% translation precision, 96.52% transition temporal precision, 83.79% transition semantic precision, and only 0.51% frame leakage and 3.38% shot leakage. The paper’s key novelty is that this camera-grid representation also enables zero-shot camera understanding from raw RGB or Canny-edge inputs, showing that the model can generalize beyond explicitly paired camera data.