Technical Summary: OmniAD

Problem Statement

Industrial anomaly analysis requires systems that not only predict the presence of defects but also localize subtle irregularities and explain their causes and implications. While Multimodal Large Language Models (MLLMs) have demonstrated strong capabilities in open-domain visual reasoning, their application to industrial anomaly analysis remains challenging. Anomalies are often subtle, category-specific, and semantically entangled with fine-grained visual evidence.

A prevalent strategy involves coupling MLLMs with external anomaly detectors (the "Expert-Aid Paradigm"), where a detector provides heatmaps or masks as visual cues for the MLLM. The authors argue this approach is brittle for two primary reasons:

1. Noisy Visual Cues: External detector outputs are often threshold-dependent and imperfect. When converted into hard cues, these errors are directly exposed to the MLLM.
2. Contextual Suppression: Even when cues roughly localize an anomaly, they can dominate the model's attention, weakening holistic reasoning about the object, surrounding structure, and failure modes.

The core issue identified is that anomaly localization and understanding are loosely coupled through externally injected cues rather than being unified within a single reasoning process.

Methodology

The authors propose OmniAD, an end-to-end framework that intrinsically integrates anomaly detection and understanding into a single multimodal reasoning process. Instead of relying on external detectors, OmniAD generates and consumes anomaly evidence within the same model and reasoning chain.

1. Multimodal Reasoning Framework

OmniAD operates through two sequential stages:

• Visual Reasoning via Semantic Anomaly Encoding (SAE):
  Rather than importing thresholded heatmaps, the model performs localization internally. Inspired by text-based segmentation, SAE reformulates anomaly detection as a text generation task.

  • The image is divided into fixed-size patches.
  • Following a one-class intuition, normal patches are discarded, and only anomalous patches are retained.
  • Run-length encoding (RLE) is applied along rows to compress contiguous anomaly patches.
  • The compressed sequence is converted into a coordinate string (e.g., <seg>(2,1),(4,2)-(4,4),...</seg>).
  • This representation is compact, information-dense, and threshold-free, serving as an intrinsic intermediate variable for downstream reasoning.

• Visual Guided Textual Reasoning (VGTR):
  The model performs high-level reasoning conditioned on the internally generated SAE evidence.

  • The SAE output acts as a semantic anchor, guiding the model to relevant regions without overriding holistic analysis.
  • The model produces a structured response containing <seg> (localization), <reasoning> (analysis of object state and defect evidence), and <answer> (final decision).
  • This design ensures anomaly evidence is treated as a semantically coherent intermediate state rather than an isolated hint, mitigating contextual suppression.

2. Joint Training Strategy

To address the scarcity of industrial training data and improve reasoning capabilities, OmniAD employs a joint post-training strategy combining Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO).

• SFT: Teaches the model the target response format, object-specific knowledge, and the coupling between localization and explanation.
• GRPO: Refines reasoning behavior using verifiable rewards without requiring expensive token-level reasoning annotations.
• Reward Functions: Three specific rewards are designed:
  1. Format Reward: Ensures the model outputs structured reasoning within specific tags.
  2. Detection Accuracy Reward: Uses the F1-score (rather than IoU) to evaluate the SAE-generated mask against ground truth, incentivizing precise localization while penalizing false positives on normal images.
  3. Answer Accuracy Reward: Evaluates the final multiple-choice answer, providing a smoother optimization signal for the reasoning path.

3. Data Curation

The authors constructed a multimodal dataset tailored for both stages. For SFT, they curated high-quality Visual Question Answering (VQA) pairs with dense supervision. For GRPO, they utilized the same data sources with answer-level supervision and verifiable localization targets, as dense reasoning annotations are unnecessary for the reinforcement learning phase.

Key Contributions

1. OmniAD Framework: A unified, end-to-end framework that intrinsically couples anomaly detection and understanding, replacing brittle external expert-aid coupling with internal anomaly evidence flow.
2. Semantic Anomaly Encoding (SAE) & VGTR: A novel mechanism where anomaly evidence is generated as a compact textual representation and immediately reused for downstream analysis, forming an intrinsic generation-consumption chain.
3. Joint Training with SFT and GRPO: A demonstration that combining supervised learning with policy optimization improves both anomaly detection accuracy and reasoning quality, validated through extensive experiments.

Experimental Results

The framework was evaluated on the MMAD dataset for anomaly understanding and four industrial anomaly detection benchmarks (MVTec-AD, VisA, MVTec-LOCO, GoodsAD).

• Anomaly Understanding (MMAD): OmniAD (7B model) achieved an average accuracy of 79.9%, outperforming strong baselines including Qwen2.5-VL-7B (by 15.9 points) and GPT-4o (by 5.0 points). It showed significant gains in Defect Classification, Localization, and Analysis, surpassing ordinary human annotators in several subtasks.
• Expert-Aid Comparison: Unlike the Expert-Aid paradigm, which often degraded performance when using external cues (e.g., heatmaps or masks), OmniAD achieved superior results by generating its own cues.
• Anomaly Detection: OmniAD delivered robust pixel-level and image-level performance across all four detection benchmarks without requiring threshold selection at inference time. It achieved the best average 0-shot pixel-level performance and competitive 1-shot results.
• Ablation Studies:
  • Training Strategy: Joint SFT+GRPO significantly outperformed either stage alone. GRPO alone failed to acquire necessary domain knowledge and structure, while SFT alone lacked the refinement for optimal reasoning.
  • Reasoning Components: Removing Visual Reasoning (SAE) or VGTR significantly degraded performance, confirming that both precise evidence generation and high-level semantic reasoning are necessary and complementary.
  • Reward Design: Both Detection and Answer Accuracy rewards were essential; the former stabilized intermediate visual evidence, while the latter ensured global consistency of reasoning and decisions.

Significance

The paper claims that strengthening intrinsic perception within MLLMs is a practical route toward accurate and interpretable industrial anomaly reasoning. By unifying localization and understanding into a single reasoning chain, OmniAD addresses the limitations of external cue injection, such as noise and attention suppression. The results suggest that for industrial applications, maintaining reasoning continuity within the model is more effective than loosely coupling external detector outputs to downstream understanding tasks. This approach offers a threshold-free, robust solution that scales across diverse anomaly types and detection benchmarks.