LVMT: Video Mask Transformer for Long-term Video Segmentation
AuthorsNarges Norouzi, Niccolò Cavagnero, Idil Esen Zulfikar, Bastian Leibe, Gijs Dubbelman, Daan de Geus
AffiliationsEindhoven University of Technology · RWTH Aachen University
LVMT makes long-term video object segmentation more capable and much faster by selectively remembering object information while training efficiently on long videos.
Key results
AP points gained on OVIS at similar inference speed.
Reduction relative to PMT on OVIS.
At least this fraction of gradient signal is retained within five-frame chunks, versus 5% without TQP.
Frames per second on an NVIDIA H100.
Over 10× faster in reported benchmark comparisons.
What the paper found
LVMT targets identity switches when objects disappear for long periods. Built on PMT with a frozen ViT encoder initialized from DINOv3, it replaces additive query propagation with a shared GRU that learns which object information to retain. Its Truncated Query Propagation strategy processes videos in five-frame chunks, carries hidden states between chunks, and truncates backpropagation at chunk boundaries, enabling longer temporal training without unbounded peak memory or vanishing gradients. On OVIS, LVMT improves mean Average Precision by 4.6 points over PMT and reduces identity switches by about 27%; gradient retention within chunks rises from 5% to at least 34%. It runs at 95 frames per second on an NVIDIA H100, close to PMT’s speed, and is over 10× faster than prior state-of-the-art systems in reported comparisons. Results across OVIS, YouTube-VIS 2022, VIPSeg, and VSPW show gains spanning instance, panoptic, and semantic video segmentation.
Original abstract
Existing online video segmentation methods struggle to track objects in long, complex videos with long-term occlusions. We hypothesize that this limitation is caused by (i) the inability of their temporal propagation mechanism to adaptively select the object information that is propagated across time, and (ii) their inability to be trained on long videos due to memory requirements and vanishing gradients. To address the first limitation, we propose to use a lightweight GRU-based temporal propagation module that can learn to select which information it keeps in memory and propagates across time. Second, to allow training on long videos, we introduce Truncated Query Propagation (TQP), a training strategy in which the model processes a video in chunks of frames, where information about tracked objects is propagated between chunks but backpropagation is only conducted in individual chunks, enabling longer temporal supervision without out-of-memory issues, inference overhead, or vanishing gradients. The resulting model is called the Long-term Video Mask Transformer (LVMT). Extensive experiments on six benchmarks show that LVMT sets a new state of the art across a range of video segmentation tasks, while retaining the speed of the highly efficient model it is based on, making it 10X faster than the prior state of the art. Code: https://www.tue-mps.org/lvmt
Read the original paperMore in Computer Vision
Browse all 61 papers →JLD: Perceptual Distance Through A Jacobian Lens
Shreshth Saini, Balu Adsumilli, Alan C. Bovik
JLD uses the Jacobian of a frozen vision model to build a faster, more resolution-robust measure of how visually different two images appear.
Tetris3D: 3D Scene Generation With Objects That Fit Together
Jaeyeong Kim, Jinhyuk Jang, Jongmin Lee, Kyehong Park, Seungryong Kim
Tetris3D generates 3D scenes whose objects physically fit together by conditioning each object on the geometry and interactions of its neighbors.
All-in-One Multilingual Scene Text Recognition with Script-aware Mixture-of-Experts
Xingsong Ye, Yongkun Du, Jiaxin Zhang, Zhixian Li, Chong Sun, Chen Li, Jing Lyu, Lianwen Jin, Zhineng Chen
A lightweight script-aware mixture-of-experts model brings more accurate, scalable multilingual scene text recognition to many languages and scripts.