NTH

Spatiotemporal Dynamics-Aware Industrial Anomaly Detection and Reasoning

AuthorsS-Wei Chen, Siangyu Yang, Yi-Chen Jan, Chun-Yi Lee, Shao-Yuan Lo

AffiliationsNational Taiwan University · {b11902027, b12902055, b11902057, cylee, sylo}@csie.ntu.edu.tw

November 14, 2026 2 min read
Watch on YouTube
The one-line take

MUSTARD helps multimodal AI spot and explain manufacturing defects revealed by unusual motion, not just by how a product looks.

Key results

68.1%
Phys-AD average AUROC

MUSTARD’s average detection score across Phys-AD categories.

57.2%
IPAD AUROC

MUSTARD’s score on the IPAD benchmark.

0.456
PAEval

Anomaly-explanation quality score on Phys-AD.

0.534
Flow-guided keyframe-selection AUROC

Ablation score, compared with 0.509 for uniform sampling.

8
Keyframes per video

Fixed keyframe budget selected using optical-flow magnitude.

30
Normal reference videos

Normal-only videos used to extract reference patterns.

What the paper found

Industrial defects can be caused by motion—a bearing that fails to rotate, for example—even when individual frames look normal. MUSTARD addresses this problem with a training-free, few-normal-shot framework that uses Dual TV-L1 optical flow to select 8 informative keyframes and create complementary views: raw frames, motion-trajectory overlays, and a curve of global flow magnitude. It compares these descriptions against normal patterns extracted from 30 reference videos, then produces an anomaly score and a natural-language explanation. The system uses Qwen2.5-VL-7B for visual descriptions and Gemma3-12B for pattern reasoning and evidence fusion; GPT-4o is used to judge explanation quality. On Phys-AD, MUSTARD reaches 68.1% average AUROC, compared with 60.3% for O-VAD, and scores 0.456 on PAEval. On IPAD, it achieves 57.2% AUROC, also ahead of the tested baselines. An ablation shows that flow-guided keyframe selection raises AUROC from 0.509 with uniform sampling to 0.534. The results suggest that explicit motion signals and comparisons with normal behavior help multimodal language models detect and explain industrial failures that static-frame reasoning can miss.

Original abstract

Industrial Anomaly Detection (IAD) is a long-established problem that aims to detect defects in industrial products during manufacturing for quality control. Most existing studies focus on static visual patterns as the basis for detection. However, in real manufacturing environments, many anomalies are dynamics-based, such as irregular movements that cannot be perceived from static cues alone. We observe that even recent Multimodal Large Language Model (MLLM)-based approaches fall short in identifying such dynamic anomalies due to limited temporal modeling capability. To address this, we introduce MUSTARD: MUltimodal SpatioTemporal Anomaly Reasoning and Detection, a framework that equips MLLMs with fine-grained spatiotemporal dynamics awareness for detecting and reasoning about dynamics-based defects. MUSTARD introduces optical-flow-guided keyframe selection and augmented modalities to capture critical motion irregularities through local motion patterns and global dynamic evolution. Its normal pattern-based reasoning pipeline requires only a few normal reference samples as the detection basis, avoiding any model updates. This enables efficient adaptation across various products and provides high decision interpretability. MUSTARD is the first MLLM-based framework for dynamics-based IAD. Experiments demonstrate its superior performance in detecting spatiotemporal anomalies and providing plausible explanations. Our code is available at https://github.com/racercode/Mustard

Read the original paper

More in Multimodal AI

Browse all 79 papers →
01Multimodal

ALIVE: Making Streaming Video Models Interactive

Jinzhao Li, Yuan Xu, Jinxuan Wu, Yukun Wang, Yijia Lei, Hanyu Ji, Yinuo Chen, Zining Huang, Chi Zhang, Hang Ye, Yiwu Zhong, Honglei Yan, Panwang Pan, Yizhou Wang, Yin Li, Miao Liu

ALIVE aims to make video-streaming AI an interactive partner that can monitor events, coordinate tasks, and ask for feedback.

Read analysis