🤖 AI

AnomalyClaw: A Universal Visual Anomaly Detection Agent via Tool-Grounded Refutation

AnomalyClaw is a training-free visual anomaly detection agent that enhances the reliability of vision-language models across diverse domains by transforming anomaly judgment into a multi-round, tool-grounded refutation process that systematically verifies candidates against normal-sample references.

Original authors: Xi Jiang, Yinjie Zhao, Zesheng Yang, Feng Zheng

Published 2026-05-12

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Xi Jiang, Yinjie Zhao, Zesheng Yang, Feng Zheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a quality control inspector at a factory, a doctor looking at an X-ray, or a satellite operator scanning the earth. Your job is to spot something "wrong" (an anomaly) in a picture. The problem is, what counts as "wrong" changes completely depending on the job. A tiny scratch on a car part is a disaster, but a scratch on a granite countertop might just be a natural texture. A new building in a satellite photo is a major change, but a new tree might just be normal growth.

For years, computer programs struggled to do this because they had to be re-trained from scratch for every single new job. If you trained a program to find cracks in bridges, it would get confused when you showed it a medical scan.

This paper introduces AnomalyClaw, a new "AI agent" that doesn't need re-training. It's like a super-smart, adaptable inspector who can jump from checking bridges to checking brains without ever needing a new textbook.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Gut Feeling" Trap

Standard AI models are like people who rely too much on their "gut feeling" (prior knowledge). If you show them a weird pattern, they might say, "That looks like a normal rock texture," because they've seen rocks before. But in the world of anomaly detection, that "weird pattern" might actually be a critical crack. They are too confident in their general knowledge and ignore the specific evidence right in front of them.

2. The Solution: The "Devil's Advocate" Agent

Instead of letting the AI make a single, snap judgment, AnomalyClaw acts like a detective holding a court trial.

The Suspect List: First, the AI looks at the image and says, "I think I see a scratch here, and maybe a dent there." These are its "suspects."
The Refutation (The Core Innovation): Instead of just confirming its own suspicions, the AI is programmed to try to prove itself wrong. It acts as a "Devil's Advocate."
- It grabs a "tool" (like a magnifying glass or a side-by-side comparison tool) to look at the "scratch."
- It compares the scratch to a pile of "perfectly normal" reference photos.
- It asks: "Is this scratch actually just a normal shadow? Does it look like the scratches in the normal photos?"
- If the evidence shows the scratch is normal, the AI refutes its own suspicion and removes it from the list.
- If the evidence shows the scratch is unlike anything in the normal photos, the suspicion stays.

This process happens in a loop (up to 5 rounds). The AI keeps trying to disprove its own findings until it is left with only the things that truly cannot be explained by "normal" examples.

3. The Toolbox: 13 Specialized Tools

To do this, AnomalyClaw has a toolbox of 13 different "tools" it can pick up, depending on the job:

Side-by-Side: It crops the suspicious area and puts it right next to a normal area to compare them pixel-by-pixel.
Expert Heatmap: It asks a specialized, pre-trained computer model (an "expert") to highlight where anomalies usually hide.
Reference Retrieval: It searches a library of normal images to find the best matches for comparison.
Zoom & Rotate: It zooms in on tiny details or rotates images to see if a "defect" is just a weird angle.

The AI is smart enough to know which tools work for which job. It won't try to use a "road crack detector" on a "brain scan."

4. The "Self-Evolving" Rulebook (Optional)

The paper also describes a cool optional feature where the AI can learn on the fly without a teacher.

Sometimes, the AI's "gut feeling" (Direct Judgment) and its "Devil's Advocate" (Refutation) will disagree. One says "It's broken," the other says "It's fine."
When this happens, the system pauses and asks a second AI (a "Reflector") to read the notes from that disagreement.
The Reflector writes a text rule (like a sticky note) for the future: "Hey, in this specific type of image, if you see a bright spot, it's usually normal, not a defect."
The next time the AI sees a similar image, it reads the sticky note and gets it right.
Crucially: It does this without anyone telling it the correct answer (no "oracle labels"). It learns purely from its own internal confusion.

5. The Results: A Universal Inspector

The researchers tested AnomalyClaw on 12 completely different worlds:

Industrial factories (finding scratches on metal).
Retail (finding torn labels on products).
Medical (finding tumors in MRI scans).
Infrastructure (finding cracks in concrete).
Remote sensing (finding new buildings in satellite photos).
Road safety (finding obstacles on highways).

The Outcome:

Better Accuracy: On almost every single test, AnomalyClaw was significantly better than just asking the AI for a quick opinion. It improved accuracy by a large margin (up to nearly 8 percentage points on some models).
No Training Needed: It worked on all these different fields without needing to be re-trained or fine-tuned for each one.
Reliability: It reduced the number of "false alarms" (calling a normal thing broken) and "missed defects" (calling a broken thing normal).

Summary Analogy

Imagine you are trying to spot a fake painting.

Old AI: Looks at the painting and says, "It looks like a real Van Gogh because it has swirls!" (It relies on general knowledge).
AnomalyClaw: Looks at the painting, suspects the swirls are fake, then pulls out a magnifying glass, compares the brushstrokes to a verified Van Gogh, checks the canvas texture, and asks an art historian bot. It only declares it a fake if it cannot find any evidence that matches a real Van Gogh. If it can't prove it's fake, it assumes it's real.

The paper claims this "refutation" approach makes the AI much smarter, more reliable, and ready to work in any industry immediately, without the expensive and time-consuming process of re-training.

Technical Summary: AnomalyClaw

Problem Statement

Visual Anomaly Detection (VAD) is critical for industrial inspection, medical imaging, and infrastructure monitoring. However, existing approaches face two primary challenges:

Domain Fragmentation: Anomaly definitions, data modalities, and annotation standards vary significantly across domains (e.g., industrial defects vs. medical lesions). Models trained in one domain rarely transfer effectively to others.
Limitations of Vision-Language Models (VLMs): While pre-trained VLMs offer cross-domain potential, their single-inference judgments are often unreliable. They rely heavily on general prior knowledge rather than fine-grained evidence from normal-sample references. Consequently, they may produce high-confidence but incorrect judgments (e.g., mistaking a natural texture for a defect or vice versa) because they lack a mechanism to explicitly verify hypotheses against specific normal references.

Methodology: AnomalyClaw

The authors propose AnomalyClaw, a training-free, backbone-agnostic VAD agent that reframes anomaly detection as a multi-round refutation process rather than a single-pass classification.

Core Architecture

The system operates on a frozen VLM backbone and consists of three concurrent components:

Refutation Agent (Main Innovation):
- Process: Instead of accumulating evidence to confirm an anomaly, the agent engages in a refutation loop. It proposes candidate anomalous features and attempts to disprove them by comparing them against normal reference images.
- Tool Library: The agent utilizes a 13-tool catalog including visual verification (side-by-side cropping, zooming), reference retrieval (DINOv2 similarity), expert probing (frozen SubspaceAD models), and domain knowledge lookups.
- Adaptivity: Tools are gated by domain applicability clauses (e.g., pixel-level diff tools are disabled for non-aligned domains like medical imaging). The agent selects tools based on a cost-ordered ladder (L1–L7) to resolve uncertainty.
- Trajectory: The agent runs for up to $K=5$ turns. In each turn, it selects a suspect feature, gathers tool-grounded evidence, and issues a verdict (found_in_ref, not_found, or inconclusive), updating the anomaly score accordingly.
Direct VLM Scorer (Co-signal):
- A single-pass, descriptor-free VLM call that provides a coarse anomaly score ( $s_D$ ). This serves as a robust safety net, particularly for backbones where the multi-turn refutation might struggle.
Fusion:
- The final score is a fixed weighted average: $s_{final} = 0.5 \cdot s_D + 0.5 \cdot s_R$ . This blend is applied uniformly across all backbones without domain-specific tuning.

Verbalized Self-Evolution (Optional Extension)

Addressing the lack of labeled positive data in cross-domain VAD, the authors introduce an Online Self-Refinement (OSR) mechanism:

Mechanism: The system monitors the disagreement between the Direct branch and the Refutation branch. If the difference $|s_D - s_R| > 0.30$ , the item is queued.
Rule Generation: Once a domain-specific queue fills (N=10 items), a "reflector" VLM analyzes the trajectories (without ground truth labels) and generates corrective textual rules.
Application: These rules are retrieved via RAG and prepended to subsequent queries, allowing the agent to "learn" from its own internal inconsistencies online.

Key Contributions

First Universal VAD Agent: AnomalyClaw is the first training-free, backbone-agnostic agent that frames VAD as compositional perception with explicit refutation. It uses a single recipe across diverse VLM backbones (GPT-5.5, Seed2.0-lite, Qwen3.5-VL-27B).
Refutation over Accumulation: The paper argues that anomaly detection is inherently comparative. By casting each inference turn as an explicit attempt to refute a candidate against normal references, the agent avoids the confirmation bias common in accumulation-style loops.
Label-Free Self-Evolution: The OSR extension demonstrates that agents can improve via internal branch disagreement without oracle labels, a capability distinct from standard prompt engineering or supervised fine-tuning.
CrossDomainVAD-12 Benchmark: A new benchmark unifying 12 distinct domains (industrial, medical, remote sensing, etc.) into a single reference-based AUROC protocol to expose backbone- and domain-dependent failure modes masked by single-domain evaluations.

Experimental Results

Evaluated on CrossDomainVAD-12 (1,418 test items across 12 domains):

Performance Gains: AnomalyClaw consistently outperforms single-step Direct inference across all backbones:
- GPT-5.5: +6.23 percentage points (pp) macro AUROC.
- Seed2.0-lite: +7.93 pp macro AUROC.
- Qwen3.5-VL-27B: +3.52 pp macro AUROC.
- All gains are statistically significant (95% confidence via stratified paired bootstrap).
Cross-Domain Generalization: The agent significantly outperforms specialized, fine-tuned VLMs (e.g., AD-Copilot, IAD-R1) on cross-domain tasks. While fine-tuned models often collapse on non-industrial domains (e.g., dropping to near-random on remote sensing), AnomalyClaw maintains robust performance without task-specific training.
Self-Evolution: The OSR extension yields a mean gain of +2.09 pp on Qwen3.5-VL-27B, comparable to a baseline using 10 oracle labels per domain for rule learning, but achieved with zero oracle labels.
Behavioral Analysis: Stronger backbones (GPT-5.5) utilize more reasoning turns and tools effectively. The refutation trajectory acts as a safety net, compensating for specific domain failures (e.g., recovering performance on D11 GI endoscopy where the refutation-only ablation failed).

Significance and Claims

The paper claims that agentic refutation fundamentally improves the anomaly understanding and reasoning capabilities of VLMs, rather than merely aggregating tool outputs.

Training-Free Adaptability: The system offers a "ready-to-use" solution for any new domain scenario without the cost of collecting large-scale anomaly annotations or retraining models.
Agent-Native Improvement: The verbalized self-evolution mechanism demonstrates that agents can bootstrap their own rulebooks from internal disagreement, a capability that label-based loops cannot replicate in zero-shot settings.
Complementary Error Correction: The fusion of a coarse Direct scorer with a detailed refutation agent creates a robust system where the Direct branch prevents catastrophic failures in domains where the multi-turn agent might over-refine or regress.

The authors position AnomalyClaw not just as a performance boost, but as a paradigm shift toward compositional perception agents that can transfer across backbones, domains, and tasks without parameter updates.