Vidu S2: Real-Time Interactive, Editable, and Spatial Video Generation

Vidu S2 targets the gap between offline video generators such as Sora and Veo and genuinely interactive visual systems. It combines Vidu S2-Avatar, an audio-visual Diffusion Transformer for continuously generated digital characters, with Vidu S2-Editing, a causal streaming editor for style transfer, virtual try-on, subject replacement, and background replacement. Avatar generation reaches 720p at up to 42 FPS, accepts reference images dynamically during a stream, and follows large-motion instructions such as dancing. Its central training contribution is Self-Replay Forcing, which re-noises detached autoregressive rollouts and replays them through a gradient-connected causal pass, improving robustness to accumulated streaming errors; temporally ordered dense captions, preference optimization, and a one-step latent super-resolution Refiner further support control and fidelity. Editing uses frame-aligned attention so each output frame reads only its corresponding source frame while reference-image tokens propagate appearance consistently, preserving input motion and timing. An inference stack based on SageAttention, SpargeAttention, Sparse-Linear Attention, W8A8 GEMM, kernel fusion, CUDA Graphs, and Ulysses-style multi-GPU parallelism makes deployment practical on lower-cost GPUs. On StreamAV-Bench, Vidu S2-Avatar scores 3.370 for visual quality and 0.617 for audio-visual synchronization, outperforming listed baselines including Runway, HeyGen, and PixVerse in complementary evaluations. Vidu S2-Editing trains on 800,000 curated clips and achieves 4.26 on the joint OpenVE–RefVIE benchmark and 9.9515 VFIDI on ViViD virtual try-on. The system also converts generated or edited streams into synchronized left- and right-eye spatial video for VR headsets.