Spatiotemporal Dynamics-Aware Industrial Anomaly Detection and Reasoning
AuthorsS-Wei Chen, Siangyu Yang, Yi-Chen Jan, Chun-Yi Lee, Shao-Yuan Lo
AffiliationsNational Taiwan University · {b11902027, b12902055, b11902057, cylee, sylo}@csie.ntu.edu.tw
MUSTARD helps multimodal AI spot and explain manufacturing defects revealed by unusual motion, not just by how a product looks.
Key results
MUSTARD’s average detection score across Phys-AD categories.
MUSTARD’s score on the IPAD benchmark.
Anomaly-explanation quality score on Phys-AD.
Ablation score, compared with 0.509 for uniform sampling.
Fixed keyframe budget selected using optical-flow magnitude.
Normal-only videos used to extract reference patterns.
What the paper found
Industrial defects can be caused by motion—a bearing that fails to rotate, for example—even when individual frames look normal. MUSTARD addresses this problem with a training-free, few-normal-shot framework that uses Dual TV-L1 optical flow to select 8 informative keyframes and create complementary views: raw frames, motion-trajectory overlays, and a curve of global flow magnitude. It compares these descriptions against normal patterns extracted from 30 reference videos, then produces an anomaly score and a natural-language explanation. The system uses Qwen2.5-VL-7B for visual descriptions and Gemma3-12B for pattern reasoning and evidence fusion; GPT-4o is used to judge explanation quality. On Phys-AD, MUSTARD reaches 68.1% average AUROC, compared with 60.3% for O-VAD, and scores 0.456 on PAEval. On IPAD, it achieves 57.2% AUROC, also ahead of the tested baselines. An ablation shows that flow-guided keyframe selection raises AUROC from 0.509 with uniform sampling to 0.534. The results suggest that explicit motion signals and comparisons with normal behavior help multimodal language models detect and explain industrial failures that static-frame reasoning can miss.
Original abstract
Industrial Anomaly Detection (IAD) is a long-established problem that aims to detect defects in industrial products during manufacturing for quality control. Most existing studies focus on static visual patterns as the basis for detection. However, in real manufacturing environments, many anomalies are dynamics-based, such as irregular movements that cannot be perceived from static cues alone. We observe that even recent Multimodal Large Language Model (MLLM)-based approaches fall short in identifying such dynamic anomalies due to limited temporal modeling capability. To address this, we introduce MUSTARD: MUltimodal SpatioTemporal Anomaly Reasoning and Detection, a framework that equips MLLMs with fine-grained spatiotemporal dynamics awareness for detecting and reasoning about dynamics-based defects. MUSTARD introduces optical-flow-guided keyframe selection and augmented modalities to capture critical motion irregularities through local motion patterns and global dynamic evolution. Its normal pattern-based reasoning pipeline requires only a few normal reference samples as the detection basis, avoiding any model updates. This enables efficient adaptation across various products and provides high decision interpretability. MUSTARD is the first MLLM-based framework for dynamics-based IAD. Experiments demonstrate its superior performance in detecting spatiotemporal anomalies and providing plausible explanations. Our code is available at https://github.com/racercode/Mustard
Read the original paperMore in Multimodal AI
Browse all 79 papers →ALIVE: Making Streaming Video Models Interactive
Jinzhao Li, Yuan Xu, Jinxuan Wu, Yukun Wang, Yijia Lei, Hanyu Ji, Yinuo Chen, Zining Huang, Chi Zhang, Hang Ye, Yiwu Zhong, Honglei Yan, Panwang Pan, Yizhou Wang, Yin Li, Miao Liu
ALIVE aims to make video-streaming AI an interactive partner that can monitor events, coordinate tasks, and ask for feedback.
Let Samples Sing in Chorus: Mitigating Spurious Correlations via Cross-view Consensus
Weiwei Li, Junzhuo Liu, Tong Chu, Hengfu Yu, Yinjie Lei, Wen Li
CHORD combines clues from multiple representation spaces to help models rely less on misleading shortcuts and perform better on underrepresented groups.
LightTAR: Light and Fast Target-Aware Representations for Multimodal Tabular Learning
Eilam Shapira, Alan Arazi, Shoham Grunblat, Roi Reichart
LightTAR adds a small set of target-aware features to standard embeddings, helping tabular models make better predictions with little extra runtime.