Seeing the Evidence, Missing the Answer: Tool-Guided Vision-Language Models on Visual Illusions

1. Problem Statement

Vision-Language Models (VLMs) exhibit a systematic failure when confronted with classic optical illusions (e.g., Ebbinghaus, Müller-Lyer, Ponzo). Even when images are counterfactually modified to remove the perceptual effect (making the stimulus objectively unambiguous), VLMs overwhelmingly predict the illusion is still present.

The authors argue this is not merely a weakness in visual representation but a training-data imbalance. Pre-training corpora contain almost exclusively genuine illusions, leading models to learn a strong prior: "images that look like an illusion always exhibit the effect." Consequently, direct visual evidence is overridden by this prior, resulting in near-systematic false positives.

The DataCV 2026 Challenge (Tasks I and II) formalizes this problem:

• Task I: Binary classification (Yes/No) on modified classic illusions where the effect may or may not hold.
• Task II: Multiple-choice questions on real-world visual anomalies and impossible scenes.
• Constraint: Inference-only. No model fine-tuning or weight updates are permitted; solutions must rely on prompting, routing, and tool usage.

2. Methodology

The authors propose a Tool-Guided Inference Framework built around a Gemini Flash-family VLM. The system operates without training, relying on a "ReAct-style" loop where the agent iteratively calls tools to generate visual evidence before answering.

Key Architectural Components:

1. Immutable Image Resource Versioning:

   • Unlike standard pipelines where tools modify an image in place, every tool call creates a new, immutable image resource (e.g., img_001, img_002) appended to a persistent registry.
   • This allows the model to branch its reasoning (e.g., drawing a line on one version while cropping a region on another) and revisit any prior annotated state. This mimics immutable value semantics in functional programming applied to visual reasoning.

2. Generic Tool Set:
   Instead of hard-coding specific modules for every illusion type (as in VisProg), the framework uses a small library of general-purpose operators:

   • Shared Core (Tasks I & II): Draw lines/rectangles/circles, crop regions, and side-by-side comparison.
   • Task II Extensions: Grid overlay, channel extraction, color sampling, color isolation, and blur.

3. Prompt-Based Routing Strategy:

   • The system prompt contains a routing mechanism that classifies the incoming question into a perceptual category (e.g., "size comparison," "line straightness," "hidden content").
   • Based on this classification, the prompt prescribes a specific sequence of tool calls. This replaces the need for a hard-coded module library, allowing the system to generalize to structurally unfamiliar illusion variants without code changes.

3. Key Contributions

• Training-Free Framework: A tool-guided annotation framework for open-ended perceptual VQA that operates entirely via zero-shot prompting.
• Immutable Resource System: An image versioning system that accumulates annotated views as named assets, enabling complex, multi-step reasoning chains with full history access.
• Prompt-Routing Design: A strategy that maps question categories to tool-use strategies within the system prompt, offering a flexible alternative to rigid module libraries.
• Empirical Analysis: Identification of three specific failure modes in VLMs regarding illusion detection (detailed below).

4. Experimental Results

The system was evaluated on the DataCV 2026 Challenge validation and test sets using Gemini Flash models.

• Generalization: The framework demonstrated strong cross-structural generalization. Performance remained consistent between the validation set and a test set containing structurally unfamiliar variants (e.g., Mach Bands rotated from vertical to horizontal). This success is attributed to the prompt describing conceptual procedures (e.g., "crop around adjacent interfaces") rather than hardcoded pixel operations.
• Tool Usage Patterns:
  • Task I: Heavily relied on the geometric core (compare crops, draw line, crop).
  • Task II: Selectively activated analysis-oriented tools (grid, color sampling, blur) only when the question category demanded it.
• Performance: The approach achieved high accuracy without any model fine-tuning, effectively mitigating the "illusion bias" by forcing the model to inspect generated evidence.

5. Critical Observations & Discussion

The authors identified three systematic phenomena in VLM behavior that warrant further research:

1. Systematic Positive-Detection Bias:

   • VLMs exhibit a near-degenerate prior to predict "illusion present" regardless of visual evidence.
   • Cause: Likely rooted in training data imbalance (lack of counterfactually modified negative examples), rather than inherited human-like perceptual vulnerabilities.

2. Spatial Precision vs. Logical Inference Dissociation:

   • Models are highly precise at spatial reasoning (e.g., drawing a perfectly straight reference line or accurately cropping a region).
   • However, they fail at logical inference over their own outputs. Even when the generated annotation clearly contradicts the illusion (e.g., a curved line next to a straight reference), the model often ignores the visual evidence and reverts to its prior belief that "Hering-looking lines are straight."

3. Sensitivity to Compression Artifacts:

   • JPEG compression and low bit-depth rendering introduce noise (blocking artifacts, quantization noise) at region boundaries.
   • These artifacts interact poorly with the model's calibration:
     • They can cause false positives in boundary detection (interpreting noise as a line).
     • They can cause false negatives in color tasks (interpreting quantization noise as a real color difference).
   • The model lacks robust thresholds to distinguish between "real" differences and compression artifacts.

6. Significance

This paper challenges the notion that VLMs simply "fail" at illusions. Instead, it reveals a dissociation between perceptual precision and inferential reliability. The models can see and annotate the evidence correctly but fail to update their beliefs when that evidence conflicts with their training priors.

The proposed solution demonstrates that tool-guided inference can recover performance without architectural changes or retraining. The findings suggest that improving VLM robustness on counterfactual stimuli requires:

1. Targeted Data Augmentation: Specifically including negative/modified illusion instances in training data.
2. Reasoning Frameworks: Utilizing immutable, multi-step reasoning chains to force models to confront visual evidence against their priors.

This work provides a blueprint for handling "hard" visual reasoning tasks where standard VLM priors are misleading, emphasizing the power of external tools and structured reasoning over raw model capacity.