Goku: A Million-Scale Universal Dataset and Benchmark for Instruction-Based Video Editing

Goku, developed by the University of Science and Technology of China and Tencent Hunyuan, pushes instruction-based video editing beyond narrow appearance transforms by releasing a 2 million-pair dataset that explicitly covers basic edits, camera movement, subject movement, reference-guided editing, and 2-to-5-task multi-edit compositions. The paper’s synthesis pipeline starts from 1 million Koala-36M clips, then uses Gemini2.5-Pro for instruction generation and progressive filtering, Grounded-SAM2 for temporal masks, VACE, Flux, Minimax-Remover, Wan2.2, and RecamMaster for task-specific synthesis, and discards about 88% of synthesized candidates to preserve alignment, temporal stability, and photorealism. On top of this data, the authors propose Goku-Edit, a dual-branch model built on Wan2.2-5B with a frozen Qwen3VL-8B MLLM text encoder, a dedicated mask-prediction branch, RoPE-aligned spatial cross-attention, and SpatialCFG for stronger structural control. They also introduce Goku-Bench, a human-verified benchmark with 1,000 test cases and 7 metrics spanning physical rule fidelity, spatial accuracy, instruction following, and task-specific subject motion, camera motion, and style transfer. On Goku-Bench, Goku-Edit delivers up to an 8% gain in instruction following over other open-source models, and ablations show that Goku scaling, filtering, and the spatial modules all contribute measurably to the final performance.