AnyTalk: Speech Animation for Arbitrary Characters Leveraging a Video Generation Model

AnyTalk converts audio into 3D speech animation for arbitrary rigged characters without paired audio-animation training data. It first adapts the Hallo audio-driven video diffusion model using Character-specific Fine-tuning, or CsF: rendered blendshape images are paired with zeroed audio embeddings, while temporal, audio-attention, and motion-attention modules remain frozen and only spatial residual layers adapt to the character. At inference, nonzero speech generates a character-consistent talking-head video with pose suppressed and lip motion emphasized. A second stage lifts this 2D motion into 3D by optimizing blendshape parameters against 14 talk-related landmarks, using expression-invariant landmark filtering, homography warping, asymmetric mouth-opening loss, and L1 regularization. Tests cover 5 characters with varied mesh structures and styles, using LibriSpeech audio; across 150 animations, AnyTalk achieves an LSE-D of 11.304 and LSE-C of 3.155, outperforming ScanTalk, DiffSpeaker plus NFR, and CodeTalker plus NFR. The method requires 20 minutes of CsF training on an NVIDIA A6000, but frame-wise optimization takes 3.12 seconds. To remove that bottleneck, AnyTalkRT distills the pipeline with feature matching and blendshape reconstruction losses, reaching 110 FPS, or 9.09 ms per frame, with some lip-sync degradation. The same CsF and landmark-optimization pipeline also transfers to MEMO, although the system remains dependent on frontal views and predefined blendshapes.