LiveFoley: Dynamic Real-Time Video-to-Audio Generation for Immersive Interactive Worlds
AuthorsYusheng Dai, Vladimir Iashin, Kangdi Wang, Yuxuan Jiang, Ilpo Viertola, Weiqiang Wang, Qiuhong Ke, Min Zhao, Jianfei Cai
AffiliationsMonash University Nanjing University ShengShu Technology Tampere University · Alibaba Token Hub, Alibaba Group Tsinghua University †Corresponding author
LiveFoley generates synchronized sound as video streams, aiming to make interactive worlds feel more immediate without sacrificing audio quality.
Key results
Hours of video used to train LiveFoley.
Look-ahead enabled by the Trailing Sliding Window.
Measured on a NVIDIA A800 GPU with the smallest hop.
Frontend computation reduction versus the smallest hop.
LiveFoley-48k temporal desynchronization score.
Listener rating out of 5 for LiveFoley’s audio-visual alignment.
What the paper found
LiveFoley generates synchronized audio continuously from live video, targeting game engines, AR/VR, and camera feeds. Unlike turn-based systems such as OpenAI GPT-Realtime, it must keep sound aligned with each visual event while operating at low latency. Its key innovation, a Trailing Sliding Window, adapts the frozen Synchformer segment encoder for streaming without retraining: it preserves the full temporal context while limiting frontend look-ahead to 0.08 s. A Unified Varhop Framework lets one model switch between hop sizes, trading delay for computation; at the largest hop, frontend computation falls 4× versus the smallest. On the audio side, History LocDiT uses past audio patches to improve denoising without added delay, while optional Future-Warmup LocDiT conditions on an approximate future patch for further quality gains. Trained on 2,042 hours of video, LiveFoley achieves 156 ms end-to-end latency on a NVIDIA A800 GPU. On VGGSound, its temporal desynchronization score is 0.48, close to offline MMAudio-L’s 0.47, and listeners rate its audio-visual alignment 3.58 out of 5. Audio quality is also assessed using Meta’s Audiobox Aesthetics metric. Together, these results make LiveFoley a practical streaming sound engine, though its audio-generation quality still trails the strongest offline models.
Original abstract
Streaming video-to-audio (V2A) generation is essential for immersive interactive applications. However, real-time constraints leave existing streaming V2A methods far behind offline models: (1) visual frontends shrink their receptive fields to cut look-ahead latency, severely degrading audio-visual alignment; (2) causal autoregressive diffusion backends denoise each audio patch in isolation, impairing audio quality. To this end, we present LiveFoley, a streaming causal autoregressive diffusion architecture that addresses both bottlenecks with imperceptible latency. For the visual frontend, we introduce a Trailing Sliding Window that converts non-causal segment-level encoders into streaming modules without retraining. By simply realigning the window, it preserves precise audio-visual alignment, while a configurable hop size reduces the frontend look-ahead well below the human perceptual threshold. Building on this tunable hop size, we propose the Unified Varhop Framework to accommodate scenarios with diverse latency and computation budgets. For the audio backend, we explore widening the denoising window of the local diffusion transformer (LocDiT) head in two temporal directions. History LocDiT adds past context to improve quality at no latency cost and serves as our default. Alternatively, Future-Warmup LocDiT breaks the autoregressive deadlock with an approximate future condition, further improving generation quality at the cost of an acceptable backend look-ahead. Extensive experiments demonstrate that LiveFoley matches offline models in temporal alignment and leads streaming systems in generation quality. Project page: https://dalision.github.io/livefoley-page/
Read the original paperMore in Multimodal AI
Browse all 79 papers →ALIVE: Making Streaming Video Models Interactive
Jinzhao Li, Yuan Xu, Jinxuan Wu, Yukun Wang, Yijia Lei, Hanyu Ji, Yinuo Chen, Zining Huang, Chi Zhang, Hang Ye, Yiwu Zhong, Honglei Yan, Panwang Pan, Yizhou Wang, Yin Li, Miao Liu
ALIVE aims to make video-streaming AI an interactive partner that can monitor events, coordinate tasks, and ask for feedback.
Let Samples Sing in Chorus: Mitigating Spurious Correlations via Cross-view Consensus
Weiwei Li, Junzhuo Liu, Tong Chu, Hengfu Yu, Yinjie Lei, Wen Li
CHORD combines clues from multiple representation spaces to help models rely less on misleading shortcuts and perform better on underrepresented groups.
LightTAR: Light and Fast Target-Aware Representations for Multimodal Tabular Learning
Eilam Shapira, Alan Arazi, Shoham Grunblat, Roi Reichart
LightTAR adds a small set of target-aware features to standard embeddings, helping tabular models make better predictions with little extra runtime.