🤖 AI

Bidirectional Multimodal Prompt Learning with Scale-Aware Training for Few-Shot Multi-Class Anomaly Detection

The paper proposes AnoPLe, a lightweight bidirectional multimodal prompt learning framework with scale-aware training that achieves state-of-the-art few-shot multi-class anomaly detection by eliminating the need for class-specific textual descriptions or external modules while effectively capturing shared normal patterns across diverse categories.

Original authors: Yujin Lee, Sewon Kim, Daeun Moon, Seoyoon Jang, Hyunsoo Yoon

Published 2026-03-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yujin Lee, Sewon Kim, Daeun Moon, Seoyoon Jang, Hyunsoo Yoon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a quality control inspector at a massive factory. This factory doesn't just make one thing; it makes everything from screwdrivers to sneakers to circuit boards.

Your job is to spot the "bad" items (defects). But here's the catch:

You are new: You've only seen one perfect example of each item (a "few-shot" scenario).
You have no manual: You don't have a list describing what a "broken screw" or a "scratched sneaker" looks like.
You need to be fast: You can't stop the assembly line to think too hard.

This is the exact problem the paper AnoPLe solves. Here is how it works, explained simply.

The Old Way: The "Over-Prepared" Inspector

Previous AI models tried to solve this by asking an expert to write a detailed description for every possible defect.

The Problem: If the factory suddenly starts making "rubber ducks" instead of "sneakers," the expert has to write a whole new manual. If the expert forgets to describe a specific type of scratch, the AI misses it. It's like trying to memorize a dictionary for every language in the world just to spot a typo. It's slow, expensive, and brittle.

The New Way: AnoPLe (The "Intuitive" Inspector)

The authors created a system called AnoPLe. Instead of memorizing a list of defects, it learns to understand the concept of "normal" and "weird" by talking to itself in two languages at once: Text and Vision.

Here are the three magic tricks it uses:

1. The "Two-Way Conversation" (Bidirectional Prompting)

Imagine you are trying to describe a "broken chair" to a friend who has never seen one.

The Text Side: You say, "It's a chair." (This gives the general idea).
The Vision Side: You show a picture of a chair with a missing leg.
The Magic: In AnoPLe, the "Text" and "Image" parts of the AI talk to each other constantly.
- The Text says: "Remember, this is a chair."
- The Image says: "But look, this specific chair has a missing leg."
- They refine each other. The text helps the image understand the category, and the image helps the text understand the specific flaw. They don't need a pre-written list of "broken chairs"; they figure it out by comparing the idea of a chair against the reality of the image.

2. The "Zoom Lens" (Scale-Aware Training)

Sometimes a defect is huge (a giant crack), and sometimes it's tiny (a speck of dust).

The Problem: Standard AI looks at the whole picture (the "Global View") and might miss the tiny dust. Or it looks too close (the "Local View") and misses the big picture.
The Solution: During training, AnoPLe is shown the object whole AND zoomed in on small patches.
The Analogy: Think of it like a security guard who learns to spot a thief by looking at the whole crowd and zooming in on individual faces.
The Cool Part: When it's time to actually work (inference), it only needs to look at the whole picture. It's like the guard has already "memorized" the details of the faces during training, so they don't need to zoom in anymore. They can spot the thief instantly from a distance.

3. The "Consistency Check" (Alignment Loss)

The AI makes two guesses:

"Is this whole image weird?" (Global guess).
"Is this specific pixel weird?" (Local guess).

The Problem: Sometimes the AI thinks the whole image is fine, but a specific pixel is broken, or vice versa.
The Solution: AnoPLe forces these two guesses to agree. If the "pixel" says "I'm broken," the "whole image" must acknowledge that. It's like a teacher checking a student's homework: "You said the answer is 5, but your calculation shows 7. Let's fix that so they match." This makes the AI much more reliable.

Why is this a Big Deal?

It's Fast: It doesn't need to run heavy, slow calculations or look up a massive database of defect descriptions. It's like a seasoned worker who just "knows" something is wrong immediately.
It's Flexible: If you introduce a brand new product (like a "rubber duck") that the AI has never seen before, it can still spot defects because it understands the concept of "duck-ness" and "broken-ness" without needing a new manual.
It Works Everywhere: The paper tested it on factory parts, and it even worked on medical scans (like MRI brain images). It's like a mechanic who can fix cars, but also happens to be great at fixing watches because they understand the concept of "gears" and "broken parts" generally.

The Bottom Line

AnoPLe is a smart, lightweight AI that learns to spot defects by having a conversation between "what the object is called" and "what the object looks like." It doesn't need a manual, it doesn't need thousands of examples, and it doesn't get confused when the factory changes products. It's the ultimate "learn on the fly" inspector.

1. Problem Statement

The paper addresses the Few-Shot Multi-Class Anomaly Detection (Few-Shot MCAD) problem, a critical challenge in real-world industrial inspection.

Context: Industrial settings involve numerous product categories with frequent updates, making it impractical to collect large, curated datasets for every category.
Constraints:
- Few-Shot: Only a handful of "normal" samples are available per category for training.
- Multi-Class: A single model must handle diverse object categories simultaneously.
- Data Scarcity: Defect patterns vary wildly across categories, while "normality" is a shared concept.
Limitations of Existing Methods:
- Description-Dependence: Current Vision-Language Model (VLM) approaches (e.g., PromptAD, IIPAD) rely on handcrafted textual descriptions of specific defects (e.g., "broken fabric"). These are hard to scale, require expert knowledge, and cause semantic confusion when collapsed into a unified multi-class space.
- Computational Cost: Methods using heavy auxiliary modules (like Q-Formers) or sliding-window inference are computationally expensive and slow for real-time deployment.
- Semantic Confusion: Aggregating heterogeneous categories without proper semantic grounding leads to overlapping feature representations, reducing detection accuracy.

2. Methodology: AnoPLe

The authors propose AnoPLe, a lightweight, scalable framework that eliminates the need for handcrafted anomaly descriptions and external modules. It leverages a pre-trained CLIP model with a novel prompt learning strategy.

Core Components:

Bidirectional Multimodal Prompt Learning:
- Textual Prompts: Instead of describing defects, the text branch uses class names as semantic anchors (e.g., "screw", "bottle") combined with a unified [abnormal] token. This avoids the need for category-specific defect descriptions.
- Visual Prompts: The visual branch learns instance-level cues directly from image evidence to capture subtle deviations.
- Bidirectional Coupling: The framework employs learnable linear projections ( $f_{v \to t}$ $f_{v \to t}$ and $f_{t \to v}$ $f_{t \to v}$ ) to fuse textual and visual context vectors at every layer of the encoder.
  - Mechanism: Class semantics guide visual features toward class-aware structures, while instance-level visual cues refine textual representations with defect-relevant details. This creates class-conditioned, defect-agnostic representations.
Scale-Aware Prompt Learning:
- Problem: Standard CLIP struggles with fine-grained localization (pixel-level) because it focuses on global semantics.
- Solution: A learnable scale-aware prefix ( $c$ ) is introduced. During training, the model sees both global images and local crops. The prefix encodes the resolution identity, allowing the model to learn unified representations that respect both global context and local details.
- Inference Efficiency: At test time, only the global prefix is used, maintaining full-image compatibility without the computational overhead of multi-scale inference.
Lightweight Decoder & Alignment:
- A lightweight decoder refines CLIP's patch embeddings to generate pixel-level anomaly maps.
- Alignment Loss ( $L_{align}$ ): A cosine similarity loss propagates local anomaly evidence (from pixel logits) to global features. This ensures consistency between pixel-level and image-level predictions, strengthening the model's ability to detect anomalies under extreme few-shot constraints.
Training Strategy:
- Pseudo-Anomalies: Since true anomalies are unavailable, the model simulates anomalies via pixel-space perturbations and latent-space noise injection.
- Memory Bank: During inference, a visual memory bank of normal features is used to compute similarity-based anomaly scores, fused with the prompt-guided scores via a harmonic mean.

3. Key Contributions

Unified Framework: Proposes AnoPLe, the first framework to solve few-shot MCAD without relying on expert-crafted anomaly descriptions or heavy auxiliary modules (like Q-Formers).
Bidirectional Interaction: Introduces a novel bidirectional prompt learning mechanism where class semantics and instance-level cues mutually refine each other, effectively organizing heterogeneous categories without semantic confusion.
Scale-Aware Training: Designs a scale-aware prefix and alignment loss that enables fine-grained localization and global consistency without increasing inference cost.
Scalability & Generalization: Demonstrates that class names alone provide sufficient inductive bias to handle diverse categories and unseen anomaly types, outperforming complex baselines.

4. Experimental Results

The method was evaluated on three major industrial benchmarks: MVTec-AD, VisA, and Real-IAD, under 1-shot, 2-shot, and 4-shot settings.

Performance:
- MVTec-AD (1-shot): Achieved 94.5% Image-AUROC and 90.8% Pixel-AUROC.
- VisA (1-shot): Achieved 86.0% Image-AUROC and 87.5% Pixel-AUROC.
- Real-IAD (1-shot): Achieved 81.2% Image-AUROC and 88.8% Pixel-AUROC.
- AnoPLe consistently outperformed or matched state-of-the-art methods (including IIPAD, PromptAD, and INP-Former) while being significantly more efficient.
Efficiency:
- Achieved inference speeds of 27.92 FPS (MVTec), 26.90 FPS (VisA), and 29.64 FPS (Real-IAD), surpassing methods requiring sliding windows or multiple forward passes.
Generalization:
- Unseen Classes: In leave-one-class-out experiments, AnoPLe showed significantly less performance degradation on unseen categories compared to vision-only baselines (INP-Former).
- Unseen Anomalies: Unlike description-based methods (PromptAD) which fail when specific defect descriptions are removed, AnoPLe remained stable when encountering novel defect types.
Medical Domain Transfer:
- Tested on medical datasets (Brain MRI, Liver CT, Retinal OCT) with 4-shot settings. AnoPLe achieved top or near-top performance, demonstrating robust cross-domain transferability without domain-specific tuning.

5. Significance

Practicality for Industry: By removing the dependency on expert-crafted defect descriptions and heavy modules, AnoPLe offers a deployable solution for dynamic industrial environments where new products and defect types emerge frequently.
Paradigm Shift: It shifts the focus from "describing defects" to "learning shared normality structures" via class names, proving that lightweight semantic anchors are sufficient for complex multi-class anomaly detection.
Efficiency: The architecture achieves state-of-the-art accuracy with minimal computational overhead, making it suitable for real-time industrial inspection systems.
Cross-Domain Potential: The successful application to medical imaging suggests the framework's underlying mechanism (learning shared normality via multimodal prompts) is universally applicable beyond industrial manufacturing.