🤖 AI

ArtifactLens: Hundreds of Labels Are Enough for Artifact Detection with VLMs

ArtifactLens is a multi-component framework that leverages the inherent knowledge of pretrained Vision-Language Models (VLMs) through in-context learning and instruction optimization to achieve state-of-the-art artifact detection using only a few hundred labeled examples per category.

Original authors: James Burgess, Rameen Abdal, Dan Stoddart, Sergey Tulyakov, Serena Yeung-Levy, Kuan-Chieh Jackson Wang

Published 2026-02-11

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: James Burgess, Rameen Abdal, Dan Stoddart, Sergey Tulyakov, Serena Yeung-Levy, Kuan-Chieh Jackson Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are an art critic tasked with spotting "glitches in the Matrix"—those tiny, creepy mistakes in AI-generated images, like a person having six fingers or a hand melting into a table.

Currently, to train an AI to be a good critic, you have to show it tens of thousands of "bad" images. It’s like trying to teach a student to spot counterfeit money by making them look at a mountain of fake bills. It’s exhausting, expensive, and by the time they learn, the counterfeiters have already changed their style.

This paper introduces ArtifactLens, a smarter way to teach AI to spot these glitches using only a few hundred examples instead of tens of thousands.

Here is how they do it, explained through three simple analogies:

1. The "Specialist Squad" (The Architecture)

Instead of hiring one giant, overwhelmed professor to look at an entire painting and say, "Is this bad?", ArtifactLens builds a Specialist Squad.

Imagine a crime scene investigation. Instead of one detective looking at everything, you have a Fingerprint Expert, a Footprint Expert, and a Face Expert. Each specialist gets a magnifying glass (a "crop tool") to zoom in specifically on the area they care about. If the Fingerprint Expert finds a smudge, the whole squad reports a "crime" (an artifact). By breaking the big problem into tiny, specific jobs, the AI doesn't get distracted by the beautiful background and misses the tiny error.

2. The "Spot the Difference" Game (In-Context Learning)

Usually, we teach AI by changing its "brain" (fine-tuning). ArtifactLens doesn't change the brain; it just gives the AI a better cheat sheet.

Imagine you’re playing a game of "Spot the Difference." Instead of explaining the rules of the game for an hour, I just show you two pictures: one where the person has five fingers, and one where they have six. You instantly "get it."

The researchers call this Counterfactual Demonstrations. They show the AI pairs of images that are almost identical, except one has a tiny glitch. This "A vs. B" comparison helps the AI realize, "Oh! The only thing that matters here is that extra finger," without needing a massive textbook of rules.

3. The "Confidence Coach" (Text Optimization)

AI models are often "too polite" or "too shy." When asked to find errors, they tend to be very cautious, thinking, "I'm not 100% sure that's a deformed hand, so I'll just say it's fine." This makes them miss almost all the mistakes.

ArtifactLens uses an LLM (like a coach) to rewrite the instructions for the AI. Instead of a boring instruction like "Find errors," the coach provides a "Full Spectrum" of instructions. It tells the AI:

"Be a strict judge: only flag it if it's undeniably broken."
"Be a paranoid detective: if something looks even slightly weird, flag it!"

By testing a whole range of "personalities" (from cautious to aggressive), the system finds the perfect "voice" that catches the most mistakes without being too dramatic.

The Bottom Line

ArtifactLens proves that you don't need a massive library of data to create an expert. By using a specialized team, smart visual comparisons, and better instructions, they created a system that is more accurate than the "big brain" models, while using 90% less training data. It’s the difference between memorizing an entire encyclopedia and simply learning how to use a magnifying glass.

Technical Summary: ArtifactLens

ArtifactLens: Hundreds of Labels Are Enough for Artifact Detection with VLMs

1. Problem Statement

As AI-generated imagery (AIGC) becomes increasingly realistic, detecting subtle "artifacts"—such as distorted hands, extra limbs, or warped object morphology—is critical for benchmarking generators and training reward models. Current state-of-the-art (SOTA) methods rely on fine-tuning Vision-Language Models (VLMs) or object detectors on massive datasets (30k–300k labeled images). This approach is computationally expensive and lacks agility; whenever a new generator evolves or new artifact types emerge, the entire labeling and fine-tuning process must be repeated. The authors identify several reasons for the difficulty of this task: the visual smallness of artifacts, biases in VLMs against rare classes, and a lack of artifact-specific knowledge in post-training.

2. Methodology

The authors propose ArtifactLens, a "scaffolding" architecture that avoids weight fine-tuning in favor of black-box optimization of frozen, pretrained VLMs. The system is built using the DSPy framework and consists of three core pillars:

Multi-Component Architecture (Specialists): Instead of asking a single VLM to detect all errors, ArtifactLens decomposes the task into $K$ independent "specialists" (e.g., a "deformed hand" specialist, a "missing leg" specialist). Each specialist uses a crop tool (via GroundingDino) to zoom into specific regions of interest, improving the VLM's ability to see small, fine-grained details. The final prediction is a logical OR of all specialist outputs.
In-Context Learning (ICL) with Counterfactual Demonstrations: To improve the VLMs without changing weights, the system uses ICL by providing image-label pairs as demonstrations. The authors introduce Counterfactual (CF) Demonstrations: instead of random examples, they retrieve pairs of images that are semantically similar (via CLIP) but have opposite labels (one with an artifact, one without). This helps the VLM isolate the specific visual features that constitute an error.
Text Instruction Optimization with Full-Spectrum Prompting: The system uses an LLM to iteratively rewrite the VLM's text instructions based on performance on a validation set. To combat the tendency of LLMs to generate "cautious" prompts (e.g., "only flag if you are 100% sure"), which exacerbates the VLM's inherent bias against the artifact class, the authors propose Full-Spectrum Prompting. This instructs the LLM to generate a diverse pool of prompts covering a wide range of confidence thresholds—from aggressive/high-recall ("flag even if uncertain") to conservative/high-precision ("only flag if undeniable").

3. Key Contributions

ArtifactLens Framework: A novel VLM scaffolding approach that achieves SOTA performance using orders of magnitude less data than fine-tuning.
Counterfactual Demonstrations: A new method for multimodal ICL that uses semantically similar but label-distinct pairs to clarify task boundaries.
Full-Spectrum Prompting: A technique to overcome VLM decision-boundary biases by optimizing instructions across a spectrum of confidence levels.
Multi-Benchmark Evaluation: The first comprehensive evaluation of artifact detection across five distinct human artifact benchmarks, testing for cross-dataset generalization.

4. Results

Data Efficiency: ArtifactLens achieves an overall F1 score of 0.82 (using Gemini-2.5-Pro), outperforming the best fine-tuned models (F1 $\approx$ 0.75) while using only 10% of the data. Even with only 200 training samples, performance degrades by only 9% compared to the full-data version.
Performance Gains: Compared to zero-shot VLMs, the scaffolding improves F1 scores by at least 45%.
Generalization: Unlike many fine-tuned models that struggle when moving to new datasets, ArtifactLens demonstrates strong generalization across different benchmarks and artifact types (object morphology, animal anatomy, etc.).
Human Comparison: The system performs comparably to (and in some metrics, better than) human annotators, who exhibit moderate subjectivity in artifact detection.

5. Significance

ArtifactLens represents a paradigm shift from model-centric (fine-tuning weights) to system-centric (optimizing workflows and prompts) AI development. It proves that the knowledge required to detect complex visual errors is already latent within large pretrained VLMs; the challenge is not "teaching" the model, but "eliciting" the correct behavior through intelligent scaffolding. This approach is highly scalable and cost-effective, making it ideal for the rapidly evolving landscape of generative AI, where new models and error types emerge weekly.