Unified Video Dense Prediction from Disjoint Data

In a paper from Yihong Sun and colleagues at Adobe Research and Cornell University, UniD tackles a central problem in scene understanding: dense labels are fragmented across incompatible image and video datasets. Built on Stable Diffusion v2, the model predicts eight properties—depth, surface normals, semantic segmentation, instance boundaries, human parts, albedo, shading, and materials—without co-annotated images or pseudo-labels. UniD first trains task specialists on their respective datasets, then distills their latent representations into one shared video U-Net through lightweight task projectors. Extended Self-Attention supplies historical-frame context, while temporal gradient matching transfers stability even to tasks without video supervision. On out-of-distribution tests, UniD reduces albedo WHDR to 0.207 versus 0.289 for the DINOv3-H baseline, and reaches 94.6 shading precision at 70 percent recall. Latent reconstruction is weaker for classification, but projector-only fine-tuning raises Cityscapes semantic mIoU from 37.8 to 58.4. The unified model runs at 1.56 FPS with a 16-frame memory bank and cuts estimated training memory from 650GB to 78GB compared with per-pixel distillation. Overall, the work shows that diffusion-model visual priors can bridge disjoint domains while improving out-of-distribution robustness, temporal coherence, and cross-task alignment, although segmentation still needs specialized fine-tuning.