🤖 AI

MMVIAD: Multi-view Multi-task Video Understanding for Industrial Anomaly Detection

This paper introduces MMVIAD, the first continuous multi-view video dataset and benchmark for industrial anomaly detection, and proposes VISTA, a model trained via a novel two-stage post-training pipeline that significantly outperforms existing video MLLMs and human-level baselines across four key industrial inspection tasks.

Original authors: Xiran Zhao, Jing Jin, Yan Bai, Zhongan Wang, Yifeng Sun, Yihang Lou, Xuanyu Zhu, Tao Feng, Yingna Wu

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xiran Zhao, Jing Jin, Yan Bai, Zhongan Wang, Yifeng Sun, Yihang Lou, Xuanyu Zhu, Tao Feng, Yingna Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a quality inspector at a factory. Your job is to spot tiny cracks, scratches, or dents on products moving down the assembly line.

The Problem: The "One-Second Glance" Trap
Currently, most computer programs designed to help with this job are like inspectors who only get to look at a product for a split second from a single angle. They take a snapshot, make a guess, and move on.

But in the real world, inspecting a product is more like walking around a sculpture. A crack might be invisible from the front, faint from the side, but glaringly obvious when you look at it from above. If a computer only sees one angle, it might miss the defect entirely, or worse, it might guess the right answer without actually "seeing" the evidence.

The Solution: MMVIAD (The "360-Degree Video Tour")
The authors of this paper, MMVIAD, have built a new training ground for computers. Think of it as a giant library of 2-second video clips where the camera spins around an object (about 120 degrees) to show every side.

Instead of just asking, "Is this broken?", this new system asks four connected questions, like a detective solving a case:

Is there a problem? (Anomaly Detection)
What kind of problem is it? (Defect Classification - e.g., is it a scratch or a hole?)
What object are we looking at? (Object Classification)
Exactly when in the video did the problem become visible? (Visible-Time Localization)

This last question is the game-changer. It forces the computer to point to the exact moment in the video where the evidence appears, rather than just guessing.

The Dataset: A "Digital Sandbox"
To build this, the researchers didn't just film real factories (which is messy and hard to control). Instead, they created a digital sandbox. They used 3D computer models to render thousands of objects (like bottles, helmets, and vases) with different materials (metal, plastic, wood).

They programmed the camera to spin around these objects and "carve" digital defects into them. Because they controlled the computer rendering, they knew exactly when a defect was visible and when it wasn't. This allowed them to create perfect "answer keys" for the videos, something that is nearly impossible to do with real-world footage.

The Test: Humans vs. Robots
They put this new system to the test against:

Human experts: Who are very good at spotting these issues.
Current AI models: The smartest video-understanding computers available today.

The Results: The "Smart Guess" Gap
The results showed a big gap. Even the best AI models struggled. They were okay at saying, "Yes, that's a helmet," or "Yes, there's a defect," but they were terrible at two things:

Pinpointing the defect type: They couldn't reliably tell a scratch from a dent.
Timing the evidence: They often couldn't say when in the video the defect was visible. They were essentially "hallucinating" the answer without seeing the proof.

The Fix: VISTA (The "Two-Step Training")
To fix this, the authors trained a new model called VISTA using a two-step method:

Step 1: The "Structured Thinking" Lesson (PS-SFT): Before teaching the AI to learn on its own, they showed it examples of how to think. They taught it to break its answer down: "First, look at the whole object. Then, look for the specific broken part. Then, figure out when you saw it." This gave the AI a good foundation.
Step 2: The "Reward Game" (VISTA-GRPO): Then, they played a game with the AI.
- If it guessed the defect type without first correctly identifying that a defect existed, it got no points (a "semantic gate").
- If it guessed the time interval correctly, it got bonus points.
- If it made up a time interval where no defect was visible, it got penalized.

The Outcome
After this training, the VISTA model got much better. On a test with objects it had never seen before, its score jumped from a failing grade (45.0) to a passing grade (57.5), beating even the most advanced commercial AI models available at the time.

In Summary
This paper introduces a new way to train computers to inspect products. Instead of just looking at a static photo, it teaches them to watch a video, spin around the object, and prove when and where they saw the problem. While current AI is still far behind human inspectors, this new training method helps the AI start to think more like a human detective, linking what it sees to exactly when it saw it.

Technical Summary: MMVIAD

Problem Statement

Industrial anomaly detection is critical for manufacturing quality control, yet existing benchmarks primarily rely on static images or sparse multi-view observations. These datasets fail to capture the continuous inspection processes found in real industrial scenarios, where cameras move around parts or parts rotate on fixtures. Consequently, structural defects (e.g., cracks, holes, bulges) often exhibit viewpoint-dependent visibility: a defect may be invisible from one angle, weakly visible from another, and clearly diagnosable only during a specific temporal interval of the inspection trajectory. Current models struggle to ground defect semantics to the specific viewpoint intervals where evidence is available, often predicting correct labels without being temporally grounded in the diagnostic video.

Methodology

1. MMVIAD Dataset and Benchmark

The authors introduce MMVIAD (Multi-view Multi-task Video Industrial Anomaly Detection), the first continuous multi-view video dataset for industrial anomaly detection.

Construction: The dataset is built using controllable object-centric rendering trajectories (Blender) based on the Anomaly-ShapeNet 3D assets. This approach allows for precise, verifiable annotations of defect visibility, which are difficult to obtain from real-world videos.
Scale and Scope: It contains over 4,000 inspection clips (2-second duration) covering approximately 120 degrees of camera motion. The dataset spans 48 object categories, 14 environments, and 6 structural anomaly types (crack, scratch, concavity, bulge, broken, hole).
Annotation Strategy: For each anomalous object, two aligned videos are generated: an anomaly-unmarked video and an anomaly-marked video (defective regions highlighted in red). By comparing these frame-by-frame and manually verifying, the authors derive precise "visible-time" intervals ( $[t_{start}, t_{end}]$ ).
Task Coupling: The benchmark evaluates four coupled tasks via Question-Answering (QA):
1. Anomaly Detection: Binary classification of defect existence.
2. Defect Classification: Categorizing the defect type (conditional on detection).
3. Object Classification: Identifying the object category.
4. Anomaly Visible-Time Localization: Identifying the temporal interval where visual evidence for the defect is present.

2. VISTA: Two-Stage Post-Training Pipeline

To address the challenges of coupled tasks and sparse rewards, the authors propose VISTA, a reference model trained via a two-stage pipeline:

Stage 1: PS-SFT (Perception-Structured Supervised Fine-Tuning): To mitigate the "cold-start" problem of reinforcement learning, the model is first fine-tuned on a teacher-synthesized dataset. A teacher model (Gemini 3.1 Pro) generates reasoning traces that decompose responses into global perception, localized segment perception, reasoning, and structured answers. This initializes the model with perception-structured reasoning behavior.
Stage 2: VISTA-GRPO (Visibility-grounded Industrial Structured Temporal Anomaly Group Relative Policy Optimization): This stage refines the model using Group Relative Policy Optimization (GRPO) with a specialized reward function:
- Format & Answer Rewards: Ensure adherence to the structured output format and correctness of anomaly detection/object classification.
- Semantic-Gated Defect Reward: Enforces a dependency where defect classification is only rewarded if the anomaly detection is correct, preventing the model from exploiting defect priors without detecting anomalies.
- Visibility-Aware Temporal Reward: Decouples the presence of visible evidence from boundary precision. It rewards correct empty predictions for normal samples, penalizes missed visible defects or hallucinated intervals, and uses a modified IoU with discovery bonuses for correct interval predictions.

Key Contributions

Dataset and Benchmark: MMVIAD is presented as the first continuous multi-view video dataset for industrial anomaly detection, featuring visible-time annotations for viewpoint-dependent structural defects and a multi-task QA evaluation framework.
Diagnostic Evaluation: The paper provides a comprehensive evaluation of human annotators, commercial video MLLMs (e.g., GPT-5.4, Gemini 3.1 Pro), and open-source models. Results highlight a persistent gap between current models and human performance, particularly in fine-grained defect classification and temporal grounding.
Reference Post-Training: The authors propose VISTA, a two-stage baseline combining PS-SFT initialization and VISTA-GRPO refinement, demonstrating improved generalization on unseen object categories.

Experimental Results

MMVIAD-Standard (Zero-Shot): Even the strongest commercial models (e.g., GPT-5.4) lag significantly behind human experts (approx. 20+ points in average score). Models perform relatively better on coarse anomaly detection and object classification but struggle with fine-grained defect recognition and temporal localization.
MMVIAD-Unseen (Generalization): On a split with unseen object categories, the base model (Qwen3-VL-8B) achieves an average score of 45.0. The VISTA model improves this to 57.5, with significant gains in anomaly detection, temporal localization, and object classification.
Synthetic-to-Real Transfer: VISTA trained on synthetic MMVIAD data shows improved performance on real-world static image datasets (MVTec AD and VisA), suggesting that anomaly understanding learned from synthetic multi-view videos can transfer to real inspection images.
Ablation Studies: The two-stage pipeline is crucial; direct RL without SFT initialization yields limited gains. The semantic-gated defect reward and visibility-aware temporal reward are shown to be essential for balancing task performance.

Significance and Claims

The paper claims that MMVIAD exposes a specific failure mode in current video MLLMs: while they can recognize coarse object or anomaly cues, they struggle to bind defect semantics to the specific viewpoint intervals where evidence is available. By introducing a dataset with verifiable temporal annotations and a structured training pipeline, the authors aim to shift industrial anomaly understanding from static image recognition toward continuous, evidence-grounded video reasoning. The work suggests that future models must integrate object-level perception, defect semantics, and temporal grounding to be effective in real-world industrial inspection workflows. The authors remain modest, noting that the dataset is currently synthetic and of moderate scale, and that real-world deployment requires further validation against sensor noise and production-line constraints.