VMAD: Visual-enhanced Multimodal Large Language Model for Zero-Shot Anomaly Detection

1. Problem Statement

Industrial Anomaly Detection (IAD) aims to identify and localize defects in manufacturing. While traditional methods rely on unsupervised learning with abundant normal samples, they struggle with generalization to new object classes. Zero-Shot Anomaly Detection (ZSAD) addresses this by using text prompts to detect anomalies on unseen objects.

However, existing ZSAD methods face two critical limitations:

1. Closed-World Constraints: Most current methods (e.g., WinCLIP, AnomalyCLIP) rely on fixed, predefined prompts and contrastive learning within a closed semantic space. They struggle to detect unseen defect types or adapt to diverse, dynamic industrial criteria.
2. Fine-Grained Perception Gap: While Multimodal Large Language Models (MLLMs) offer open-ended text generation and flexible analysis, they often lack the fine-grained visual sensitivity required for industrial defects. Anomalies are often subtle, localized, and exhibit minimal visual differences from normal regions, leading to poor discriminability in standard MLLMs.

2. Methodology: VMAD Framework

The authors propose VMAD (Visual-enhanced MLLM Anomaly Detection), a framework that integrates a visual branch with an MLLM to achieve simultaneous anomaly localization, segmentation, and explainable text generation.

A. Core Architecture

VMAD consists of a standard MLLM (based on LLaVA) and a specialized visual branch. The system accepts image-text pairs and outputs:

• Segmentation Masks: For precise anomaly localization.
• Explanatory Text: Descriptions of defects, impact analysis, and suggestions.
• Special Token: The model generates a [seg] token to trigger the mask decoder.

B. Key Innovations

1. Defect-Sensitive Structure Learning (DSSL)
To address the lack of discriminability in MLLMs, DSSL transfers "patch-similarity" knowledge from the visual domain to the LLM space.

• Mechanism: It calculates the similarity between local image patches and global normal features in the visual branch. Simultaneously, it computes the similarity between visual patch tokens and semantic tokens (combining text and visual cues) in the LLM space.
• Alignment: A Patch-Based Similarity Distribution (PBSD) loss aligns these two distributions. This forces the LLM to learn that anomalous patches have low similarity to normal global representations, effectively transferring visual structural cues to the language model to enhance anomaly discrimination.

2. Locality-enhanced Token Compression (LTC)
Standard visual projectors often compress visual tokens too aggressively, losing fine-grained spatial details crucial for defect detection.

• Mechanism: LTC employs a coarse-to-fine injection scheme.
  • Locality-aware Downsampling: Uses deformable attention and a local-context learner to downsample features while preserving geometric variations and local context.
  • Multi-level Feature Integration: Instead of relying solely on the final layer, LTC aggregates features from multiple CLIP encoder layers. Shallow layers provide low-level details, while deeper layers provide semantic understanding.
• Benefit: This balances efficiency (reducing token count) with the preservation of rich, multi-level semantic cues necessary for detecting subtle defects.

3. Dataset: Real Industrial Anomaly Detection (RIAD)

Recognizing the lack of high-quality datasets for MLLM-based IAD, the authors introduced RIAD.

• Scale: 28,040 images across 24 object categories and 15 defect types.
• Content: Includes paired image-text data with detailed anomaly masks.
• Annotations: Beyond binary masks, it includes semantic segmentation for specific defect types. Text annotations (generated via GPT-4) provide anomaly descriptions, impact analyses, and maintenance suggestions, creating a rich resource for training reasoning capabilities.

4. Experimental Results

The authors evaluated VMAD on zero-shot benchmarks (MVTec-AD, VisA, WFDD) and the new RIAD dataset.

• Cross-Dataset Performance: On MVTec-AD, VisA, and WFDD, VMAD outperformed state-of-the-art methods (including AnomalyGPT, Myriad, and WinCLIP).
  • MVTec-AD: Achieved 95.8% Image-level AUROC and 96.1% Pixel-level AUROC.
  • Improvement: Showed significant margins (e.g., +3.3% to +3.8% over Myriad) in pixel-level and region-level analysis, attributed to the fine-grained perception of LTC and DSSL.
• Cross-Category Performance: On the RIAD dataset (training on 16 classes, testing on 8 unseen), VMAD achieved superior results, particularly in pixel-level AUC-PR (improvements of 2.0%–14.5% over other multimodal methods).
• Qualitative Analysis: VMAD successfully localized defects and provided coherent, context-aware textual explanations (e.g., identifying a loose screw and suggesting tightening it), demonstrating true zero-shot reasoning.

5. Key Contributions

1. VMAD Framework: A novel MLLM-based architecture that unifies anomaly detection, segmentation, and explainable analysis, moving beyond closed-world prompt constraints.
2. DSSL Scheme: A cross-modal learning strategy that aligns visual patch-similarity distributions with LLM semantic spaces, significantly improving the model's ability to distinguish subtle anomalies.
3. LTC Mechanism: A novel visual projector that mines multi-level local features, preserving fine-grained details while maintaining computational efficiency.
4. RIAD Dataset: A comprehensive, real-world industrial dataset with detailed semantic masks and multi-faceted text annotations (description, impact, suggestion), filling a gap in MLLM-based IAD research.

6. Significance

This work represents a paradigm shift in industrial inspection by leveraging the generative and reasoning capabilities of MLLMs. By solving the "fine-grained perception" bottleneck through DSSL and LTC, VMAD enables open-world anomaly detection where systems can not only find defects but also explain why they are defects and how to fix them. This paves the way for more adaptive, human-AI collaborative industrial processes that do not require retraining for every new product line or defect type.