GenCLIP: Generalizing CLIP Prompts for Zero-shot Anomaly Detection

This paper proposes GenCLIP, a novel framework for zero-shot anomaly detection that enhances generalization and stability through multi-layer prompting, a dual-branch inference strategy balancing category specificity with generalization, and an adaptive text prompt filtering mechanism.

The Gist

Imagine you are a quality control inspector at a massive factory. Your job is to spot defective products on a conveyor belt. In the past, you had to spend months training specifically for one type of product, like "bottles." Once you mastered spotting cracks in bottles, you were useless if the factory switched to making "cables" or "chewing gum." You'd have to start from scratch.

This is the problem with old AI anomaly detection: it's too specialized.

Enter GenCLIP, a new AI system designed to be the ultimate "universal inspector." It can look at a bottle, a cable, or a weird industrial pipe it has never seen before and instantly say, "That looks broken," without needing any prior training on that specific item.

Here is how GenCLIP works, explained through simple analogies:

1. The Problem: The "One-Size-Fits-None" Dilemma

Previous AI models tried to solve this using a "General Description." Imagine a detective who only knows the phrase: "This is a photo of a bad object."

• The Issue: While this works for some things, it's too vague. If you show the detective a specific weird part called a "pipe fryum," the phrase "bad object" doesn't help them understand the specific shape or texture of that pipe. They might miss the defect because they aren't looking closely enough at the details.

2. The Solution: GenCLIP's "Multi-Layer Detective Team"

GenCLIP improves on this by using a Multi-Layer Prompting strategy. Think of this as giving the detective a team of specialists, each looking at the object from a different distance:

• The Macro Specialist: Looks at the big picture (the overall shape).
• The Micro Specialist: Looks at the tiny details (scratches, textures, edges).
• The Semantic Specialist: Understands the concept (is this a pipe? is it metal?).

Instead of just asking the AI to look at the "final answer" (like previous models), GenCLIP asks all these specialists to weigh in simultaneously. It combines their observations to create a much richer, more detailed mental image of what "normal" and "abnormal" look like. This prevents the AI from getting confused or "overfitting" (memorizing the training data too strictly).

3. The "Filter": Cleaning Up the Confusing Names

Sometimes, factory parts have weird names like "02," "pcb1," or "pipe_fryum." If you tell an AI, "Look for a defect in 'pcb1'," the AI might get confused because "pcb1" sounds like a code, not a description of what the object is.

GenCLIP uses a Class Name Filter (CNF).

• The Analogy: Imagine you are describing a lost dog to a police officer. If you say, "It's a dog named 'Unit 42'," the officer might not know what to do. But if you say, "It's a dog," they know exactly what to look for.
• How it works: GenCLIP checks the name. If the name is confusing or just a code (like "02"), it automatically swaps it for a generic, clear word like "object." This ensures the AI focuses on the visual reality of the item, not a confusing label.

4. The "Dual-Branch" Strategy: The Best of Both Worlds

This is GenCLIP's secret sauce. Instead of relying on just one way of thinking, it runs two parallel investigations at the same time and combines their results:

• Branch A: The Detail-Oriented Detective (Vision-Enhanced)

  • This branch looks at the specific image, uses the "Multi-Layer" team, and applies the "Name Filter." It knows exactly what the object is supposed to look like.
  • Goal: Catch specific, fine-grained defects (like a tiny scratch on a specific screw).

• Branch B: The Intuitive Detective (Query-Only)

  • This branch ignores the specific name and the detailed image features. It relies purely on a "General Sense" of what a "good" thing looks like versus a "bad" thing.
  • Goal: Catch weird outliers where the specific name doesn't matter, or where the object is so strange that the AI needs to rely on pure intuition.

The Final Verdict: GenCLIP takes the report from the Detail Detective and the Intuitive Detective, blends them together, and produces a final score. This makes the system incredibly robust. If one branch misses something, the other likely catches it.

Why This Matters

Before GenCLIP, if a factory wanted to detect defects on a new product, they had to collect thousands of photos of that new product and train a new AI model. It was slow and expensive.

With GenCLIP:

1. You can point the camera at any object (even one the AI has never seen).
2. It instantly knows if it's broken.
3. It highlights exactly where the break is.

It's like upgrading from a specialized tool that only fits one screw, to a Swiss Army Knife that can fix anything, anywhere, right out of the box.

Technical Summary

1. Problem Statement

Zero-Shot Anomaly Detection (ZSAD) aims to identify anomalies in object categories that were not present during training, leveraging the generalization capabilities of Vision-Language Models (VLMs) like CLIP. However, existing approaches face significant challenges:

• Trade-off between Generalization and Specificity: Methods using highly general prompts (e.g., "A photo of a good object") often fail to capture class-specific cues, while methods relying on specific class names struggle with unseen categories or ambiguous industrial labels (e.g., "pipe_fryum" or "02").
• Overfitting: Adapting prompts to small auxiliary datasets often leads to overfitting, where the model learns dataset-specific noise rather than robust anomaly patterns.
• Ineffective Feature Alignment: Standard CLIP models are pre-trained for classification, not anomaly detection. Directly using fixed text templates often results in poor alignment between visual features and text embeddings for distinguishing normal vs. abnormal states.
• Dynamic Prompt Instability: Previous hybrid approaches (combining static and dynamic prompts) often suffer from entanglement between static general prompts and dynamic class-specific tokens during inference, reducing stability.

2. Methodology: GenCLIP

The authors propose GenCLIP, a framework designed to learn and leverage "General Query Prompts" (GQPs) more effectively through Multi-layer Prompting and a Dual-Branch Inference strategy.

A. Multi-Layer Prompting

Instead of relying solely on the final layer's class token (as in standard CLIP), GenCLIP integrates visual information from multiple layers of the frozen CLIP vision encoder.

• General Query Prompt Tokens (GQPs): Learnable tokens ($QP$) shared across all classes to represent the general concepts of "normal" and "abnormal."
• Multi-Layer Vision Prompt Tokens (MVPs): Patch features are extracted from multiple layers ($i$) of the vision encoder. These are projected via learnable linear layers (MLP) to generate vision prompt tokens ($V^i_P$).
• Augmentation: The MVPs are added to the shared GQPs to create vision-tuned tokens ($V^i_Q = QP + V^i_P$). This enriches the text embeddings with hierarchical visual context (from low-level textures to high-level semantics), preventing overfitting and capturing class-specific details without needing explicit class names in the prompt.
• Text Layer Prompt Tuning: Learnable tokens are inserted before each Transformer layer in the text encoder to refine the alignment between the augmented text prompts and the visual features.

B. Class Name Filtering (CNF)

Industrial datasets often contain ambiguous or non-descriptive class names (e.g., "pcb2", "pipe_fryum") that confuse the VLM.

• Mechanism: During the inference of the vision-enhanced branch, CNF computes the cosine similarity between the input image and two sentences: one with the original class name and one with a generic placeholder ("object").
• Action: If the image is more similar to the generic term, the specific class name is replaced with "object" in the text prompt. This ensures that only semantically meaningful inputs contribute to the vision-language alignment.

C. Dual-Branch Inference Strategy

To balance specificity and generalization, GenCLIP employs two parallel branches during inference:

1. Vision-Enhanced Branch:
   • Combines the GQPs with multi-layer MVPs and applies CNF.
   • Captures fine-grained, class-specific features and semantic context.
   • Generates score maps ($S^i_V$) based on the similarity between image patches and enriched text embeddings.
2. Query-Only Branch:
   • Uses only the highly generalized GQPs (without vision features or specific class names), utilizing the generic term "object."
   • Learns to encode general representations of normal/abnormal instances independent of specific categories.
   • Generates score maps ($S^i_Q$) that identify outliers where class information might be detrimental.

Final Aggregation: The final anomaly score map is a weighted sum of the outputs from both branches, followed by Gaussian smoothing:
$$S_{seg} = G\left( \alpha \sum S^i_V + (1-\alpha) \sum S^i_Q \right)$$
This complementary approach improves stability and reliability across unseen categories.

3. Key Contributions

1. Multi-Layer Prompting: A novel strategy that integrates visual prompt tokens from multiple CLIP layers into learnable query prompts. This mitigates overfitting and captures a broader spectrum of visual information (textures to semantics) for robust anomaly detection.
2. Dual-Branch Inference: An innovative inference mechanism that combines a vision-enhanced branch (for specificity) and a query-only branch (for generalization). This allows the model to handle both class-specific anomalies and general outliers effectively.
3. Adaptive Text Prompt Filtering (CNF): A mechanism to automatically detect and replace ambiguous or non-descriptive class names with generic terms, ensuring high-quality text-image alignment in industrial settings.
4. State-of-the-Art Performance: The framework achieves superior results without requiring task-specific fine-tuning of the backbone VLM, relying instead on efficient prompt learning.

4. Experimental Results

The authors evaluated GenCLIP on six benchmark industrial datasets: MVTec-AD, VisA, MPDD, BTAD, SDD, and DTD-Synthetic.

• Performance Metrics: The model was evaluated using Pixel-level AUROC, PRO (Per-Region Overlap), Image-level AUROC, and AP.
• Comparison: GenCLIP outperformed existing SOTA methods, including WinCLIP, AnomalyCLIP, and AdaCLIP.
  • Pixel-Level: Achieved an average 95.4% AUROC and 88.5% PRO across datasets, surpassing the second-best method by 0.8% AUROC and 6.5% PRO.
  • Image-Level: Achieved 87.3% AUROC and 87.6% AP.
• Ablation Studies:
  • Removing the multi-layer prompting (MVP) or the query-only branch resulted in significant performance drops, confirming the necessity of both components.
  • CNF was shown to improve performance specifically on classes with ambiguous names (e.g., "pipe_fryum", "pcb1").
  • t-SNE visualizations confirmed that the dual-branch strategy creates distinct but complementary feature clusters for normal and abnormal states.

5. Significance

GenCLIP addresses a critical bottleneck in industrial anomaly detection: the inability of current ZSAD models to simultaneously handle unseen object categories and ambiguous industrial labeling.

• Practical Applicability: By filtering out confusing class names and leveraging a dual-branch strategy, GenCLIP is highly suitable for real-world manufacturing environments where labeled anomaly data is scarce and object naming conventions vary.
• Robustness: The method demonstrates that learning robust general prompts through multi-layer visual integration is more effective than relying on static templates or unstable dynamic prompts.
• Efficiency: It maintains the "train-free" or "lightweight training" advantage of VLMs (only prompt tokens are learned) while significantly boosting performance to exceed fully supervised or heavy fine-tuning approaches in zero-shot settings.