AnomalyVFM -- Transforming Vision Foundation Models into Zero-Shot Anomaly Detectors

1. Problem Statement

Zero-shot Anomaly Detection (ZSAD) aims to identify and localize abnormal regions in images without access to any in-domain training images (neither normal nor anomalous). While recent state-of-the-art methods leverage Vision-Language Models (VLMs) like CLIP to transfer high-level semantic knowledge, purely Vision Foundation Models (VFMs) like DINOv2 have underperformed in this setting.

The authors argue that this performance gap stems from two practical limitations in prior VFM-based approaches:

1. Limited Data Diversity: Existing auxiliary datasets lack sufficient diversity and realistic defect coverage, which is critical for training VFMs to generalize across arbitrary object classes.
2. Shallow Adaptation Strategies: Previous methods typically only fine-tune a simple output head (e.g., an MLP) while leaving the VFM's internal visual representations unchanged. This prevents the model from learning the specific features necessary to distinguish normal vs. abnormal appearances across diverse objects.

2. Methodology: AnomalyVFM

The authors propose AnomalyVFM, a model-agnostic framework that transforms any pretrained VFM into a robust zero-shot anomaly detector. It consists of two core components:

A. Three-Stage Synthetic Dataset Generation

To overcome data diversity limitations, the authors propose a pipeline to generate a large-scale, diverse synthetic dataset without requiring any real-world training images.

1. Anomaly-Free Image Generation: Uses a modern flow-matching model (FLUX) to generate high-quality, anomaly-free images of objects based on text prompts (e.g., "A close-up photo of [Object]..."). Prompts are dynamically constructed using an LLM (GPT-4o) to select from 100 objects and 50 backgrounds.
2. Anomaly Synthesis (Inpainting):
   • A foreground mask is extracted using a salient object segmentation network (IS-Net).
   • A random rectangular region $R$ is sampled on the object.
   • The model generates an anomalous version of the image by inpainting a defect into region $R$ using a prompt describing the anomaly (e.g., "cracked," "damaged").
3. Feature-Based Filtering: Since generative models may fail to adhere to prompts, a filtering step is introduced. Features are extracted from both the normal ($I$) and anomalous ($I_a$) images using a pretrained VFM (DINOv2). A cosine distance map is computed; if the distance is below a threshold, the sample is discarded. This ensures only samples with genuine, visible defects are retained.

B. Parameter-Efficient Adaptation Mechanism

Instead of fine-tuning the entire VFM or just the head, AnomalyVFM employs a lightweight adaptation strategy:

• Low-Rank Feature Adapters (LoRA): LoRA blocks are injected into the attention mechanisms (Query, Value, and Output projections) of every transformer block in the VFM backbone. This allows the model to evolve its internal visual representations with minimal trainable parameters.
• Decoder: A lightweight convolutional decoder upsamples the adapted features to produce pixel-level anomaly segmentation maps.
• Confidence-Weighted Loss: To handle noise from imperfect synthetic labels, the segmentation loss is weighted by a confidence map predicted by the decoder. The loss function is defined as:
  $$L_{seg} = L_{base}(M_o, M_{GT}) \times C - \alpha \log(C)$$
  where $C = 1 + \exp(c)$ and $c$ is the predicted confidence. This down-weights ambiguous supervision.

3. Key Contributions

1. AnomalyVFM Framework: A general framework that converts any pretrained VFM into a competitive zero-shot anomaly detector, outperforming VLM-based methods.
2. Scalable Synthetic Data Generation: A novel three-stage pipeline that generates diverse object-defect combinations and automatically filters for quality, creating a dataset superior to existing auxiliary datasets for VFM fine-tuning.
3. Efficient Adaptation: Demonstrates that injecting LoRA adapters throughout the backbone, combined with confidence-weighted losses, is more effective than simple head fine-tuning.

4. Experimental Results

The method was evaluated on 9 industrial and 9 medical anomaly detection benchmarks.

• Performance: Using RADIO (a VFM) as the backbone, AnomalyVFM achieved an average image-level AUROC of 94.1% across 9 industrial datasets. This surpasses the previous state-of-the-art (Bayes-PFL) by 3.3 percentage points.
• Localization: It also improved pixel-level AUROC by 0.9 percentage points over the best existing methods.
• Generalization:
  • Model Agnostic: The framework significantly improved performance across different backbones (DINOv2, DINOv3, RADIO).
  • Medical Domain: Despite being trained only on industrial synthetic data, it generalized well to medical datasets (e.g., Brain MRI, Colonoscopy), demonstrating strong cross-domain capabilities.
  • Few-Shot Capability: When fine-tuned on just a few normal samples (few-shot regime), AnomalyVFM matched or exceeded the performance of dedicated few-shot methods (e.g., INP-Former) without architectural changes.
• Efficiency: Inference is significantly faster (20.5ms) compared to competitors like Bayes-PFL (208.5ms).

5. Significance and Impact

• Bridging the Gap: The paper proves that purely visual foundation models, which are inherently better suited for visual tasks than VLMs, can achieve state-of-the-art zero-shot performance if provided with diverse training data and proper adaptation strategies.
• Data-Centric Innovation: It highlights that the bottleneck in VFM adaptation is often data diversity rather than model architecture. The proposed synthetic generation pipeline offers a scalable solution to the scarcity of diverse anomaly datasets.
• Practical Deployment: The method requires no in-domain training data, making it highly suitable for industrial and medical applications where collecting anomaly data is impossible or unethical.
• Future Direction: The work suggests that future anomaly detection research should focus on improving the realism of synthetic defects and integrating multi-modal data (e.g., depth) to further enhance zero-shot capabilities.