FiLo++: Zero-/Few-Shot Anomaly Detection by Fused Fine-Grained Descriptions and Deformable Localization

FiLo++ is a zero-/few-shot anomaly detection framework that enhances performance by generating task-specific fine-grained descriptions via large language models and employing a deformable localization module with multi-scale cross-modal interaction to accurately detect and localize anomalies of varying shapes and sizes.

The Gist

Imagine you are a quality control inspector at a factory. Your job is to spot defective products on a conveyor belt.

The Old Problem:
Traditionally, to learn what a "defect" looks like, you'd need to see thousands of broken items first. But what if you've never seen a broken item before? Maybe it's a new product line (a "cold start"), or maybe you only have one or two broken examples to study. This is the challenge of Zero-Shot (no examples) and Few-Shot (very few examples) anomaly detection.

Previous AI methods tried to solve this by asking a simple question: "Is this picture 'normal' or 'abnormal'?"

• The Flaw: It's like asking a child to spot a specific type of bug in a garden by just saying, "Look for bugs." The child might get confused because "bugs" can look like leaves, dirt, or shadows. Also, if the bug is a weird shape or size, the child might miss it or think a whole bush is a bug.

The New Solution: FiLo++
The authors of this paper created a new AI system called FiLo++ (Fused Fine-Grained Descriptions + Deformable Localization). Think of FiLo++ as a super-smart detective who doesn't just look for "bugs," but knows exactly what kind of bug to look for and where to look.

Here is how FiLo++ works, broken down into two main superpowers:

1. The Detective's Notebook (FusDes)

• The Problem: Old AI used generic descriptions like "damaged" or "broken." This is too vague. A scratch on a car is different from a tear in a shirt.
• The FiLo++ Fix: Before looking at the product, FiLo++ consults a Super-Brain (a Large Language Model like GPT-4).
  • Analogy: Imagine you are looking for a lost key. Instead of saying "Look for a lost thing," the Super-Brain says, "Look for a silver key with a blue tag, possibly bent, located near the door."
  • FiLo++ generates a custom, detailed list of what could be wrong with that specific object (e.g., "a scratch on the left side," "a missing screw," "a color stain").
  • It then uses a filter to remove the confusing or irrelevant descriptions, keeping only the most accurate ones. This makes the AI's "mental checklist" much sharper.

2. The Shape-Shifting Magnifying Glass (DefLoc)

• The Problem: Once the AI knows what to look for, it needs to find where it is. Old methods tried to compare tiny square patches of the image to text.
  • Analogy: Imagine trying to find a specific irregularly shaped stain on a rug by comparing it to a rigid, square cookie cutter. If the stain is round, long, or jagged, the cookie cutter won't fit, and you'll miss it. Or, you might mistake a shadow in the background for the stain.
• The FiLo++ Fix: FiLo++ uses a Shape-Shifting Magnifying Glass.
  • Step 1 (The Guide): It first uses a tool called Grounding DINO to roughly point out, "Hey, the object is here, ignore the background." This stops the AI from getting distracted by the floor or walls.
  • Step 2 (The Position): It tells the AI, "The defect is likely in the top-left corner."
  • Step 3 (The Magic Lens): Instead of using a rigid square lens, FiLo++ uses Deformable Convolution.
    • Analogy: Imagine a lens made of soft clay. If the defect is a long, thin crack, the lens stretches out to cover the crack. If the defect is a small dot, the lens shrinks. It molds itself to the exact shape and size of the problem, ensuring it doesn't miss anything weirdly shaped.

The "Few-Shot" Bonus

If you do have a few examples of broken items (1 to 4 photos), FiLo++ uses a special trick. It takes the "rough map" from the Shape-Shifting Magnifying Glass and says, "Okay, only compare the new item to the old broken ones in these specific areas." This prevents the AI from getting confused by the rest of the image, making it incredibly accurate even with very little data.

Why It Matters

• Speed: You don't need to wait months to collect thousands of broken products to train the AI. You can start immediately.
• Precision: It doesn't just say "Something is wrong." It can tell you what is wrong (e.g., "It's a hole in the fabric") and exactly where it is, even if the hole is a weird shape.
• Versatility: It works on factory parts, medical scans (like finding tumors in MRI images), and more.

In Summary:
FiLo++ is like upgrading from a detective who only knows the word "crime" to a detective who has a detailed file on every possible type of crime, knows exactly where to look, and carries a magnifying glass that can stretch and shrink to fit any clue perfectly. This allows it to spot problems instantly, even if it has never seen that specific problem before.

Technical Summary

1. Problem Statement

Anomaly detection (AD) is critical in industrial manufacturing, medical imaging, and transportation. Traditional methods rely on one-class classification, requiring extensive normal training data, which limits their applicability in cold-start scenarios where data is scarce.
While Zero-Shot (ZSAD) and Few-Shot (FSAD) methods using multimodal models (like CLIP) have emerged, they face two primary limitations:

1. Detection Accuracy: Existing methods rely on handcrafted, generic text prompts (e.g., "abnormal") that fail to capture the diverse and specific nature of anomalies across different object categories.
2. Localization Precision: Simple patch-level image-text matching often struggles to localize anomalies with varying shapes and sizes, leading to false positives in background regions and poor handling of multi-patch anomalies.

2. Methodology: FiLo++

FiLo++ is an extension of the authors' previous work (FiLo), designed to address the above limitations through two core modules: FusDes (for detection) and DefLoc (for localization).

A. Fused Fine-Grained Descriptions (FusDes)

This module enhances the text encoder's ability to distinguish normal from abnormal states by generating highly specific prompts.

• LLM-Generated Descriptions: Instead of generic terms, the method uses Large Language Models (LLMs, e.g., GPT-4o) to generate detailed, category-specific anomaly descriptions (e.g., "cracks," "holes," "discoloration") for each object.
• Hybrid Prompt Templates: It combines fixed human-designed templates with learnable adaptive text vectors. The template structure is: A [domain] photo of [state] [cls] with [anomaly cls] at [pos].
• Runtime Prompt Filtering: To address "cross-semantic ambiguity" (where normal and abnormal text features overlap), the method calculates cosine distances between image features and all generated text prompts. It filters out prompts falling within an overlapping distance interval, retaining only the most discriminative text features for the final score calculation.

B. Deformable Localization (DefLoc)

This module improves the spatial accuracy of anomaly localization, particularly for irregular shapes and sizes.

• Initial Localization with Grounding DINO: The method uses the vision foundation model Grounding DINO to generate initial bounding boxes around foreground objects. This filters out background noise, preventing false positives in irrelevant areas.
• Position-Enhanced Text Prompts: The coordinates of the Grounding DINO bounding boxes are mapped to nine spatial regions (e.g., top-left, center). This positional information is injected into the text prompts, making them spatially aware.
• Multi-scale Deformable Cross-modal Interaction (MDCI): Instead of standard sliding windows, FiLo++ employs deformable convolutions at multiple scales. This module aggregates patch features from the CLIP image encoder using deformable kernels of varying shapes (e.g., 1x1, 3x3, 5x5, 1x5). This allows the model to adaptively capture anomalies of irregular shapes and sizes while interacting with the position-enhanced text features.
• Few-Shot Branch: For FSAD, a position-enhanced patch matching strategy is added. It uses a memory bank of normal samples but constrains the matching scope using the initial Grounding DINO localization results to improve efficiency and accuracy.

3. Key Contributions

1. FusDes Module: A novel approach that fuses LLM-generated fine-grained descriptions, fixed/learnable templates, and runtime filtering to create highly discriminative text features, significantly improving detection accuracy and interpretability.
2. DefLoc Module: A localization framework integrating Grounding DINO for background suppression, position-enhanced prompts, and a Multi-scale Deformable Cross-modal Interaction (MDCI) module. This effectively handles anomalies of diverse morphologies.
3. Unified ZSAD/FSAD Framework: FiLo++ is the first to seamlessly integrate a few-shot branch that leverages the localization priors from the zero-shot module, achieving state-of-the-art performance in both settings without extensive retraining.
4. Comprehensive Evaluation: Extensive experiments on industrial (MVTec-AD, VisA) and medical (BrainMRI, RESC) datasets demonstrate robust generalization across domains.

4. Experimental Results

The authors evaluated FiLo++ on MVTec-AD, VisA, BrainMRI, and RESC datasets.

• Zero-Shot Performance (VisA):
  • Image-level AUC: 84.5% (Previous best: 83.9% by FiLo).
  • Pixel-level AUC: 96.2% (Previous best: 95.9% by FiLo).
• Few-Shot Performance (VisA, 1-shot):
  • Image-level AUC: 88.3% (Outperforming AnomalyGPT at 87.4%).
  • Pixel-level AUC: 97.3% (Outperforming AnomalyGPT at 96.2%).
• Medical Domain Generalization:
  • On the BrainMRI dataset (1-shot), FiLo++ achieved 86.6% Image-AUC, significantly outperforming the second-best method (AnomalyGPT) by 13.5% and the domain-specific MedCLIP by 16.9%.
• Ablation Studies:
  • Removing the MDCI module or the Grounding DINO filter resulted in significant drops in Pixel-AUC, confirming the necessity of deformable convolutions and background suppression.
  • Runtime prompt filtering was shown to be crucial for separating normal and abnormal text distributions.

5. Significance

FiLo++ represents a significant advancement in anomaly detection by bridging the gap between semantic understanding (via LLMs) and spatial precision (via deformable localization).

• Interpretability: Unlike "black box" methods, FiLo++ can identify what the anomaly is (e.g., "a hole") and where it is, based on the similarity to specific text descriptions.
• Data Efficiency: It achieves state-of-the-art results with zero or very few training samples, making it ideal for cold-start industrial applications and medical diagnostics where data collection is difficult.
• Robustness: The ability to generalize from industrial datasets to medical imaging (MRI/OCT) without domain-specific retraining highlights the robustness of the proposed cross-modal interaction mechanisms.

In conclusion, FiLo++ demonstrates that integrating large language models for fine-grained semantic generation with advanced vision foundation models for deformable spatial localization creates a powerful, flexible, and interpretable framework for next-generation anomaly detection.