💻 computer science

DiffSpot: Can VLMs Spot Fine-Grained Visual Differences in Web Interfaces?

This paper introduces DiffSpot, a code-driven benchmark demonstrating that even state-of-the-art vision-language models struggle to detect fine-grained visual differences in web interfaces, achieving low recall rates even on simple changes and showing that detection difficulty varies significantly by CSS property rather than pixel magnitude or semantic distance.

Original authors: Linhao Zhang, Aiwei Liu, Yuan Liu, Xiao Zhou

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Linhao Zhang, Aiwei Liu, Yuan Liu, Xiao Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have two nearly identical photos of a webpage. In one photo, a button is blue; in the other, it's a slightly lighter blue. Or maybe the text is just a tiny bit bolder. To a human, spotting this might take a second of squinting. But for an Artificial Intelligence (AI) that "sees" images, this is like trying to find a single grain of sand that changed color on a beach.

This paper, titled DiffSpot, introduces a new test to see how good modern AI models are at spotting these tiny, specific changes on websites. Here is the breakdown in simple terms:

1. The Problem: AI is Great at the Big Picture, Bad at the Details

Current AI models (called Vision-Language Models or VLMs) are like brilliant art critics. They can look at a picture and tell you, "This is a sunny day at the beach with a family playing." They are excellent at understanding the general story.

However, they struggle with the "fine print." If you ask them, "Did the color of that specific button change from dark blue to light blue?" they often miss it. They might say, "No, everything looks the same," or they might make up a change that didn't happen (like saying the text changed when it didn't).

2. The Solution: Building a "Spot the Difference" Game with Rules

The researchers wanted to test this ability, but they couldn't just ask humans to find differences on random websites. Humans are biased; they notice big, obvious things but miss subtle ones. Also, finding two webpages that are exactly the same except for one tiny detail is nearly impossible on the real internet.

So, the team built DiffSpot, a custom-made test suite. Think of it like a video game level designer who builds a perfect "spot the difference" puzzle from scratch.

How they built it: Instead of taking screenshots and hoping for the best, they started with the computer code (HTML/CSS) that builds a webpage.
The "Mutation": They took a piece of code, changed just one tiny setting (like making a button 5% lighter), and then re-generated the image.
The "Grounding Gate": This is a quality control step. Sometimes, changing one line of code accidentally messes up the whole page layout. The researchers used a digital "gatekeeper" to check: Did the change happen exactly where we wanted it, and nowhere else? If the change spilled over to other parts of the page, they threw that test case out.

The result is a dataset of 4,400 pairs of images. Some have a tiny, specific change (like a font getting slightly bolder), and some have absolutely no change at all.

3. The Test: Putting 13 Top AIs to the Challenge

They took 13 of the smartest AI models available today (including models from Google, OpenAI, Anthropic, and others) and asked them to look at these pairs and list the differences. They didn't give the AI any hints or examples; it was a "blind test."

The Results: The AI Struggled
Even the best AI models failed to spot the majority of the changes.

The Score: The top-performing model only found about 41% of the actual changes. That means it missed roughly 6 out of every 10 changes.
The Hard Mode: When the changes were very subtle (the "Hard" tier), the models got even worse, finding less than 23% of the changes.
The Hallucination Problem: Some models were so eager to find a difference that they made things up. They claimed a button changed color when it didn't. Others were so cautious they said "no change" even when there was one.

4. The Surprising Discovery: It's About What Changed, Not Where

The researchers expected that the difficulty would depend on the type of website (e.g., maybe shopping sites are harder than news sites). They were wrong.

The "What" Matters: The difficulty depended entirely on what property was changed.
- If the AI had to spot a change in text or position (moving an item), it did okay.
- If the AI had to spot a change in gradients (color blends) or line height (spacing between text lines), it performed terribly, even if the visual difference was huge.
The "Where" Doesn't Matter: It didn't matter if the change was on a finance website or a travel blog. The AI's ability to spot the change was consistent across all topics.

5. Why This Matters (According to the Paper)

The paper concludes that while AI is getting better at understanding images generally, it is still "blind" to the specific, fine-grained details that make up a user interface.

Pixel size isn't the answer: You might think, "If the change is big enough, the AI will see it." The study showed this isn't true. Even large pixel changes in certain categories (like gradients) were missed, while tiny changes in others (like text) were caught.
The "Magic" is missing: The AI isn't just failing to see the pixels; it's failing to understand the concept of the change (e.g., "this text is bolder").

In Summary:
DiffSpot is a rigorous "spot the difference" test for AI. It reveals that even the smartest AI models today are like a person trying to read a menu in a dark room: they can tell you there's a restaurant, but they can't reliably tell you if the price of the soup changed by a few cents. The paper proves that for AI to truly master web interfaces, it needs to get much better at noticing the tiny, specific details.

Technical Summary: DiffSpot

Problem Statement

Vision-language models (VLMs) have achieved significant progress in high-level image-text alignment and language-guided reasoning. However, their ability to perceive fine-grained visual differences remains limited. This limitation is critical for systems operating on rendered web interfaces, such as GUI agents and design tools, which must verify not only that a screen changed but exactly which visual property changed, where it changed, and whether the change was confined to the intended element.

Existing benchmarks for VLMs largely lack a dedicated "spot-the-difference" regime for web interfaces. Current approaches face two bottlenecks:

Data Scarcity: Real web pages rarely provide near-identical before/after screenshots differing by only one small UI property with balanced coverage across elements, properties, and difficulty levels.
Annotation Bias: Human labeling is selective and attention-dependent, often over-representing salient or semantically meaningful changes while missing subtle, property-level visual differences.

Methodology: The DiffSpot Pipeline

The authors introduce DiffSpot, a code-driven benchmark for open-ended spot-the-difference tasks on web interfaces. The core insight is to move difference construction from image space to code space. Since web interfaces are programmatic artifacts generated from HTML/CSS, visual differences can be defined and controlled before rendering occurs.

The construction pipeline consists of five stages:

Corpus Curation:
- Seeds are drawn from Chrome User Experience Report and Majestic Top-1M domains (2M domains).
- URLs are expanded via sitemaps and filtered by HTML structure fingerprints to ensure structural uniqueness.
- Pages are rendered in headless Chromium (1280×800 viewport) and paired with self-contained HTML regenerated by an LLM to remove third-party dependencies.
- Pairs are filtered by CLIP similarity (≥0.70) to ensure the regenerated HTML matches the original visual structure.
Programmatic Mutation:
- The system applies atomic, localized mutations to a single CSS property of a single target element.
- 13 CSS-property operators are defined across four families: Typography (font_weight, font_size, letter_spacing, line_height, text), Color (color, opacity, gradient), Layout (position, spacing, justify), and Shape (border, rounded).
- Mutations are executed via Tailwind CSS class swaps or inline-style overrides.
- Difficulty Stratification: Each operator is divided into Easy, Medium, and Hard tiers based on the magnitude of the change (e.g., Tailwind scale steps for discrete properties, em-offsets for continuous properties).
Grounding Gate Validation:
- To ensure the mutation corresponds to a clean, localized visual change, a "grounding gate" validates the rendered result.
- It checks three predicates:
  1. Effectiveness: Pixel change inside the target bounding box (bbox) is non-zero.
  2. Locality: Pixel change outside the target bbox is zero (no layout reflow contamination).
  3. Selector Resolution: The target element must resolve correctly in the DOM.
- Only pairs passing all predicates are retained.
Polish and Filtering:
- Structured mutation records are converted into natural-language descriptions using an LLM.
- Content filters remove cases with text truncation, viewport overflow, or font rendering issues.
Stratified Sampling and No-Diff Control:
- The final dataset contains 4,400 image pairs:
  - 3,900 "has-diff" pairs: 100 pairs per cell across 13 operators × 3 difficulty tiers.
  - 500 "no-diff" pairs: Identical screenshots rendered twice to measure hallucination rates.

Evaluation Setup

Models: 13 frontier VLMs were evaluated zero-shot, including proprietary APIs (Gemini 3.1 Pro, GPT-5.4, Claude Opus 4.7) and open-weight models (Kimi K2.5, Qwen3.5-VL, GLM-4.6V, InternVL3.5).
Task: Models receive before/after screenshots and are asked to list all observed visual differences without examples or hints.
Metrics:
- Accuracy: (TP + TN) / Total.
- Recall: Percentage of true mutations identified (broken down by difficulty tier).
- No-Diff Accuracy (Specificity): Percentage of unchanged pairs correctly identified as having no change (1 - hallucination rate).
Judging: Responses are evaluated by an LLM judge (gpt-oss-120b) using a visual-effect-equivalence rubric that tolerates paraphrasing. Robustness checks with two other judges confirmed ranking stability.

Key Results

1. Low Ceiling on Fine-Grained Perception

Even the best-performing model, Gemini 3.1 Pro, achieved only 40.7% Recall on true visual changes.

Hard-tier Recall was below 23% for every model evaluated.
Seven of the 13 models fell below 30% overall accuracy.
Some models (e.g., InternVL3.5) barely exceeded the trivial "always-no-diff" baseline, indicating a failure to detect changes rather than just poor precision.

2. Property-Specific Failure Modes

Performance is highly dependent on what changed, not where it appeared.

High Recall Operators: justify (87.0% for Gemini 3.1 Pro), text (70.3% for Claude Opus 4.7), and opacity (58.3% for Kimi K2.5) are more easily detected.
Low Recall Operators: gradient (max 26.7%), line_height (median 4.0%), and rounded (median 13.3%) remain extremely difficult.
Domain Independence: Recall varied minimally across 15 source domains (max difference 7.1 pp), whereas variation across operators was significant. This suggests the capability gap is property-specific, not domain-specific.

3. Visual Magnitude Does Not Predict Difficulty

The paper demonstrates that neither pixel magnitude nor CLIP distance reliably predicts model performance:

Pixel Change: Operators with large pixel changes (e.g., gradient, line_height) often have low recall, while those with small pixel changes (e.g., text, position) have high recall. Correlation $r \approx -0.08$ .
CLIP Distance: CLIP embedding distances are compressed and show near-zero correlation with recall ( $r \approx +0.06$ ).
Conclusion: Models struggle to perceive and name specific CSS-level visual properties, rather than simply failing to detect large image differences.

4. Sensitivity–Restraint Trade-off

The "no-diff" pairs revealed a critical trade-off:

Aggressive Reporters: Models like GLM-4.6V-Flash hallucinated changes in 24.2% of no-diff pairs.
Overly Conservative: Models like InternVL3.5 reported almost nothing, achieving 100% specificity but near-zero recall.
Pareto Frontier: Only three closed-source models (Gemini 3.1 Pro, Claude Opus 4.7, GPT-5.4) reached the "accurate and restrained" region (Recall ≥30%, hallucination ≤5%). No open-weight model achieved this balance.

5. Scaling and Reasoning

Reasoning Mode: For the Qwen3-VL family, switching to "Thinking" mode improved performance significantly only at the larger scale (235B parameters), not at 30B.
Scaling: Increasing parameter size alone (e.g., Qwen3-VL 30B to 235B in Instruct mode) did not improve performance, suggesting that scale without specific architectural or training adjustments is insufficient for this task.

Significance and Claims

The paper claims that DiffSpot is the first benchmark for open-ended spot-the-difference on rendered web interfaces. Its significance lies in:

Methodological Shift: Moving from human-annotated image pairs to code-driven, programmatic mutation ensures controlled, balanced, and bias-free ground truth.
Diagnostic Precision: It reveals that current VLMs are brittle on fine-grained visual perception, with performance heavily dictated by the specific CSS property being mutated.
Practical Relevance: The benchmark directly addresses the needs of GUI agents and design tools, highlighting that current models cannot reliably verify localized UI changes.
Evaluation Rigor: By including "no-diff" controls, it disentangles true visual sensitivity from hallucination, exposing a sensitivity-restraint trade-off that aggregate metrics often hide.

The authors conclude that fine-grained visual difference detection on web interfaces is "far from solved," with even the strongest models missing roughly three out of every five true visual changes, particularly in hard-tier scenarios.