FadeMem: Distance-Aware Memory Consolidation for Autoregressive Video Diffusion

FadeMem, developed by researchers at Zhejiang University, UNSW, Data61/CSIRO, and Baidu Inc., is a distance-aware KV memory consolidation method for autoregressive video diffusion that targets the core bottleneck of long-rollout generation: bounded caches. The paper shows that frame correlations decay with temporal distance in a frequency-dependent way, with fine textures and local motion fading faster than scene layout and identity, and fits this behavior with an approximate power law, r*(t) ∝ t^-b. FadeMem turns that observation into a single ordered memory of at most M entries, defaulting to M = 12, where new KV blocks are inserted as fine-grained recent entries and older adjacent entries are progressively merged into coarser span-level anchors using a power-law temporal warp with β = 0.3. On the 60-second MovieGenBench setting at 480 × 832 and 16 FPS, the inference-only variant, FadeMem-TF, achieves a VBench-Long average of 80.45, while light fine-tuning lifts FadeMem-FT to 81.03, surpassing LongLive’s 80.55 and improving subject consistency, background consistency, motion smoothness, and imaging quality. In VLM-based evaluation with Gemini 3.1-Pro, FadeMem reaches the best visual stability score at 4.84. Ablations show that weighted-average consolidation outperforms nearest-state selection and max pooling, and that removing the first-frame anchor drops the average to 79.67, confirming that the method’s dense-near, sparse-far cache hierarchy is the key to preserving long-range identity and scene coherence under a fixed memory budget.