Technical Summary: VisualNeedle

Problem Statement

While frontier Multimodal Large Language Models (MLLMs) report over 90% accuracy on fine-grained perception benchmarks, these scores often mask a lack of genuine visual engagement. The paper identifies three "shortcuts" that inflate benchmark performance without requiring true active visual search:

1. Linguistic Priors: Models infer plausible answers from question wording or lexical cues without viewing the image.
2. Coarse Global Semantics: Visual encoders rely on scene-level gist, bypassing the need to resolve fine-grained local details.
3. Tool-Output Invariance: In "think-with-images" pipelines, corrupting intermediate visual outputs (e.g., replacing crops with black images) often leaves final decisions unchanged, suggesting models do not actually rely on the returned visual evidence.

The core problem is that current benchmarks fail to distinguish between models that actively search for and localize evidence versus those that rely on these shortcuts. Simply increasing image resolution or question volume does not force models to engage with local visual evidence.

Methodology

To address these limitations, the authors introduce VisualNeedle, a benchmark designed specifically to close these shortcuts.

Benchmark Design

VisualNeedle consists of 300 curated questions set in information-dense scenes (e.g., urban street views, dense documents, shelves, bookcases, and maps). The dataset is constructed around three anti-shortcut constraints:

• Question-Only Insufficiency: Answers cannot be inferred from the question text or lexical cues alone.
• One-Shot Global Insufficiency: Decisive evidence is not visible in a downsampled, full-image view; models must zoom in.
• Returned-Evidence Dependence: Success must rely on the specific visual content returned during interaction.

The benchmark evaluates five fine-grained capabilities:

1. OCR: Reading tiny or distant text.
2. Color Recognition: Identifying localized attributes.
3. Entity Recognition: Locating specific objects in cluttered scenes.
4. Spatial Relation Judgment: Reasoning about local spatial relationships.
5. Occluded Object Recognition: Inferring properties of partially hidden objects.

Evaluation Protocols

The authors evaluate 9 prominent MLLMs (including Gemini 3.1 Pro, GPT-5.4, and Qwen3-VL) across four distinct settings to isolate specific failure modes:

1. Text-Only: Models receive only the question to test linguistic priors.
2. No-Tool: Models receive the full image once (one-shot) without tools to test global perception.
3. Tool-Enabled: Models can iteratively call a Crop tool to obtain high-resolution local observations.
4. Crop-Black (Counterfactual): A diagnostic setting where the interaction trajectory is preserved, but every returned crop is replaced with an all-black image of the same size. This tests whether performance gains come from the act of calling tools or the visual evidence they return.

Key Contributions

1. Identification of Shortcut-Driven Mismeasurement: The paper translates the three identified failure modes (language priors, global semantics, and tool-output invariance) into explicit benchmark design constraints.
2. VisualNeedle Dataset: A 300-question benchmark for active visual search in information-dense scenes that requires models to locate small, decisive local evidence before answering.
3. Crop-Black Diagnostic: A novel evaluation setting that preserves tool-use protocols while removing visual content, proving that success on this benchmark hinges on genuine intermediate visual evidence rather than tool-call trajectories.

Experimental Results

The evaluation of 9 models reveals persistent limitations in current MLLMs:

• Failure of Global Perception: In the No-Tool setting, all models scored below 20% accuracy (best: 17.89%), confirming that global scene semantics are insufficient.
• Text-Only Failure: Text-only accuracy remained below 10%, demonstrating that linguistic priors cannot solve these tasks.
• Tool-Enabled Gaps: Even with tools, the best model (Gemini 3.1 Pro) reached only 56.01% accuracy, still trailing the human majority-vote baseline of 63.00% and the human Pass@3 of 74.67%.
• Evidence Dependence (Crop-Black): The Crop-Black setting showed a massive performance drop. For Gemini 3.1 Pro, accuracy fell from 56.01% (real crops) to 12.00% (black crops). In contrast, on other benchmarks like V* and HR-Bench, models maintained high accuracy (75–87%) even with black crops, indicating those benchmarks do not strictly require intermediate visual evidence.
• Search Efficiency: Analysis of tool usage revealed that high accuracy correlates with efficient search (fewer, targeted calls). Models like GPT-5.4 made many tool calls (avg. 11.14) but achieved lower accuracy (31.23%), suggesting they struggle to converge on the correct evidence or integrate it reliably.

Significance

The paper claims that VisualNeedle provides a diagnostic, shortcut-resistant testbed that exposes the fundamental limitations of current MLLMs in active visual search. The results demonstrate that:

• High benchmark scores on existing datasets often reflect shortcut exploitation rather than genuine visual reasoning.
• Current models, even the most advanced, cannot yet reliably perform the multi-step search, localization, and evidence integration required for fine-grained tasks in dense scenes.
• The Crop-Black setting serves as a critical filter to distinguish between models that truly "think with images" and those that merely simulate tool use.

The authors conclude that active visual search remains a significant challenge, and future visual agent systems must improve not just in calling tools, but in the reliability of localizing and integrating visual evidence.