AG-VAS: Anchor-Guided Zero-Shot Visual Anomaly Segmentation with Large Multimodal Models

This paper presents AG-VAS, a novel zero-shot visual anomaly segmentation framework that leverages Large Multimodal Models enhanced with learnable semantic anchor tokens, a Semantic-Pixel Alignment Module, and a specialized instruction dataset to overcome limitations in abstract concept representation and achieve state-of-the-art localization performance across industrial and medical benchmarks.

The Gist

Imagine you are a quality control inspector at a factory. Your job is to spot defects on products coming down the assembly line—scratches on a car, a hole in a fabric, or a tumor in a medical scan.

In the past, you needed a specific training manual for every single type of product. If a new type of widget arrived, you had to stop the line, study it for weeks, and then you could start checking it. This is slow and expensive.

AG-VAS is like hiring a super-intelligent, all-knowing detective who has read every book in the library and seen every type of object in the world. This detective doesn't need a manual for new products. You can just point at a picture and say, "Find the weird stuff here," and they will instantly know what a "scratch" or a "hole" looks like, even if they've never seen that specific object before.

Here is how the paper explains this magic, broken down into simple concepts:

1. The Problem: Why Old AI Gets Confused

Previous AI models (like the ones based on CLIP) are like students who are great at reading definitions but terrible at finding things in a messy room.

• The Issue: They know what a "hole" is in a dictionary, but they struggle to find a tiny hole on a specific piece of carpet because "hole" is an abstract idea, not a concrete object like an "apple."
• The Result: They often point at the wrong spot or get confused between the background and the defect.

2. The Solution: The "Anchor" System

The authors created a new system called AG-VAS. Think of this system as giving the AI detective three special "magnifying glasses" or Anchors to hold onto while they search. These are special words (tokens) added to the AI's vocabulary:

• The [SEG] Anchor (The "What"): This is the absolute anchor. It tells the AI, "Look for specific shapes like scratches, holes, or cracks." It connects the abstract idea of a defect to a real visual shape.
• The [NOR] and [ANO] Anchors (The "Compare"): These are relative anchors. They act like a balance scale.
  • [NOR] asks: "What does a normal version of this look like?"
  • [ANO] asks: "What looks different or broken compared to that?"
  • By comparing the two, the AI can spot the "odd one out" much better.

Analogy: Imagine trying to find a red marble in a pile of blue marbles.

• Old AI: "I know red is a color, but I'm not sure which one is red."
• AG-VAS: "Okay, I see the blue ones ([NOR]). Now, I'm looking for the one that isn't blue ([ANO]). Ah, there it is!"

3. The Translator: SPAM

The AI has two brains: one that understands language (the "Big Brain") and one that sees pixels (the "Eyes"). Sometimes, they speak different languages. The Big Brain says "scratch," but the Eyes see a blurry line.

The paper introduces a Semantic-Pixel Alignment Module (SPAM). Think of this as a translator or a bridge. It takes the Big Brain's idea of "scratch" and perfectly aligns it with the specific pixels on the screen, ensuring the AI knows exactly where to draw the line.

4. The Training: "Anomaly-Instruct20K"

You can't just give a detective a list of rules; they need to practice. The authors created a massive new textbook called Anomaly-Instruct20K.

• Instead of just showing pictures, this dataset teaches the AI to describe the defect before finding it.
• Example: "The fabric usually has a smooth weave. But here, there is a dark, jagged line. That is a defect."
• This teaches the AI to understand the story of the defect (what it should look like vs. what it actually looks like), making it much smarter at spotting errors.

5. The Result: The "Zero-Shot" Superpower

"Zero-shot" means the AI can do the job without ever seeing that specific object during training.

• Real-world test: The AI was trained on industrial defects (like broken cables) but was then tested on medical images (like skin tumors or colon polyps) it had never seen before.
• The Outcome: It worked amazingly well. It could look at a new medical scan, understand the instruction "find the polyp," and draw a perfect outline around it, even though it was trained mostly on factory parts.

Summary

AG-VAS is a new way of teaching AI to spot defects. Instead of just memorizing pictures, it teaches the AI to:

1. Anchor its search on specific defect types.
2. Compare the normal vs. the abnormal.
3. Translate its thoughts into precise pixel-perfect outlines.

It's like upgrading a security guard from someone who just memorizes a list of "bad guys" to a detective who understands human behavior, knows what a "normal" day looks like, and can instantly spot anything out of the ordinary, no matter where they are.

Technical Summary

1. Problem Statement

Zero-Shot Visual Anomaly Segmentation (ZSAS) aims to localize anomalous regions on objects from unseen categories without retraining. While existing methods often rely on CLIP-based models, they face two fundamental limitations:

1. Abstract Nature of Anomalies: Unlike concrete objects (e.g., "apple"), anomalies (e.g., "scratches," "holes") are abstract, context-dependent concepts lacking stable visual prototypes, making them difficult to map from text to pixels.
2. Misalignment: There is a weak alignment between high-level semantic embeddings (from Large Multimodal Models, LMMs) and low-level pixel-level spatial features, leading to imprecise localization.
3. Limitations of Existing LMMs: While LMMs like LISA offer strong reasoning, they often fail in ZSAS by confusing foreground/background or failing to produce binary segmentation masks directly, requiring heuristic post-processing.

2. Methodology: AG-VAS Framework

The authors propose AG-VAS, an end-to-end framework that integrates learnable semantic anchors into an LMM-based pipeline to bridge the gap between semantic understanding and pixel-level segmentation.

A. Core Components

1. Semantic Anchors (The Vocabulary Expansion):
   The framework introduces three learnable tokens into the LMM vocabulary to act as semantic bridges:

   • [SEG] (Absolute Anchor): Encodes world knowledge regarding the specific appearance, shape, and location of defects (e.g., "a hole," "a scratch"). It translates abstract anomaly semantics into explicit visual entities.
   • [NOR] and [ANO] (Relative Anchors): Model the contextual contrast between normal and abnormal patterns. They help the model distinguish anomalies by comparing candidate regions against surrounding normal areas.

2. Semantic-Pixel Alignment Module (SPAM):
   To address the misalignment between LMM embeddings and pixel features, SPAM uses a cross-modal attention mechanism. It takes semantic features from a semantic image encoder and pixel features from a pixel image encoder (e.g., SAM), aligning them via Multi-Head Cross-Attention (MHCA). This ensures the anchor tokens are grounded in high-resolution visual details.

3. Anchor-Guided Mask Decoder (AGMD):
   A lightweight decoder that takes the refined anchor embeddings (from the LLM) and pixel embeddings to generate masks.

   • It uses bidirectional cross-attention to iteratively update latent features.
   • It produces two types of probability maps: an absolute map from [SEG] and a contrastive map from [NOR] vs. [ANO].
   • The final binary mask is a fusion of these maps, thresholded at 0.5.

B. Training Strategy & Dataset

• Anomaly-Instruct20K: A new large-scale instruction-tuning dataset curated specifically for anomaly segmentation. It organizes anomaly knowledge into structured fields: Expectation (normal state), Observation (visual deviation), Diagnosis (reasoning), Summary (segmentation instruction with anchors), and Explanation.
• Multi-Task Learning: The model is trained jointly on:
  • Anomaly-Instruct20K (for structured anomaly reasoning).
  • Anomaly-Seg20K (raw anomaly images for contrast learning).
  • General Object Segmentation (e.g., ADE20K) and VQA datasets (to maintain general LMM capabilities).
• Training Objective: Combines Textual Autoregressive Loss (for generating text/anchors) and Segmentation Loss (BCE + Dice) for mask prediction.

C. Inference Modes

AG-VAS supports flexible interaction:

• Direct Segmentation: User asks to segment; model outputs mask directly.
• Describe-then-Segment: Model describes the anomaly before outputting the mask.
• Segment-then-Explain: Model outputs the mask and then explains the reasoning.

3. Key Contributions

1. AG-VAS Framework: A novel anchor-guided paradigm that introduces absolute ([SEG]) and relative ([NOR]/[ANO]) semantic tokens to effectively bridge LMM reasoning with pixel-level segmentation.
2. Anomaly-Instruct20K Dataset: A specialized instruction-tuning dataset that injects structured world knowledge about defects (appearance, shape, spatial attributes) into LMMs, facilitating better zero-shot generalization.
3. State-of-the-Art Performance: The method achieves consistent SOTA results across six industrial and medical benchmarks in the zero-shot setting, outperforming both CLIP-based and existing LMM-based methods.
4. Robust Rejection Capability: Unlike many LMMs that hallucinate anomalies, AG-VAS demonstrates a strong ability to reject normal images (predicting empty masks when no anomaly exists), a critical feature for practical deployment.

4. Experimental Results

• Benchmarks: Evaluated on 6 datasets: MVTec-AD, KSDD2, RSDD (Industrial) and ISIC, ColonDB, ClinicDB (Medical).
• Performance:
  • Using LLaVA-OneVision-7B, AG-VAS achieved 51.0 AP, 52.7 F1-Max, and 44.8 IoUano on MVTec-AD, surpassing all competitors.
  • It significantly outperformed CLIP-based methods (e.g., Bayes-PFL, AnomalyCLIP) in IoUano, indicating superior localization precision.
  • It showed exceptional generalization on medical datasets where no medical data was used during training.
• Rejection Accuracy: On normal samples (MVTec-AD), AG-VAS achieved an IoUnor of 87.7%, meaning it correctly identified normal images as having no anomalies, whereas other LMMs often failed (IoUnor < 10-40%).
• Ablation Studies: Confirmed that removing [SEG] or [NOR]/[ANO] significantly degrades performance. The SPAM module and the Anomaly-Instruct20K dataset were also proven essential for high accuracy.

5. Significance

AG-VAS represents a significant advancement in Zero-Shot Visual Anomaly Segmentation by leveraging the reasoning capabilities of Large Multimodal Models while overcoming their traditional weakness in precise pixel-level localization.

• Practicality: By directly outputting binary masks and effectively rejecting normal samples, it solves the "false positive" problem common in LMMs, making it viable for industrial quality control and medical diagnosis.
• Generalization: It demonstrates that injecting structured domain knowledge (via anchors and instruction tuning) allows LMMs to generalize to unseen categories and domains (e.g., medical) without specific retraining.
• Interpretability: The framework supports "Describe-then-Segment" and "Segment-then-Explain" modes, providing human-readable reasoning alongside the segmentation, which is crucial for trust in safety-critical applications.