Vision-DeepResearch Benchmark: Rethinking Visual and Textual Search for Multimodal Large Language Models

To address the limitations of existing benchmarks in evaluating multimodal large language models' visual and textual search capabilities, this paper introduces the Vision-DeepResearch Benchmark (VDR-Bench), a rigorously curated dataset of 2,000 instances designed for realistic assessment, alongside a proposed multi-round cropped-search workflow that effectively enhances visual retrieval performance.

The Gist

Imagine you are hiring a team of super-smart detectives (AI models) to solve a mystery. Your goal is to see if they can actually look at clues (images) and do research (search the web) to find the truth, or if they are just guessing based on what they already know.

This paper, titled "Vision-DeepResearch Benchmark," is basically a report card for these detective AI teams. The authors found that the current "tests" we give these AIs are broken, so they built a new, much harder test called VDR-Bench.

Here is the breakdown in simple terms:

1. The Problem: The Old Tests Were "Cheating"

The authors realized that the old tests for AI detectives had two major flaws, like a game with broken rules:

• Flaw #1: The "Text-Only" Shortcut.
  Imagine a detective is asked, "What stadium is this soccer game in?" and shown a picture of a yellow jersey.

  • The Cheat: The AI doesn't actually look at the picture. It just reads the text question, thinks, "Oh, yellow jersey = Borussia Dortmund = Signal Iduna Park," and guesses the answer. It never needed to look at the image!
  • The Reality: The AI is just reciting facts it memorized, not doing real visual detective work.

• Flaw #2: The "Perfect Match" Trap.
  Imagine the detective is asked to find a specific building.

  • The Cheat: The old test gives the AI the exact same photo of the building. The search engine instantly finds a copy of that exact photo with the name written on it. It's like asking someone to find a needle in a haystack, but then handing them the needle on a silver platter.
  • The Reality: In the real world, you rarely find the exact same photo. You have to look at a blurry part of a crowd, zoom in, and piece together clues. The old tests didn't simulate this messiness.

2. The Solution: VDR-Bench (The "Real World" Test)

To fix this, the authors built VDR-Bench, a new test with 2,000 tricky questions. Think of it as a survival course for AI detectives.

• No Cheating Allowed: They designed questions where you must look at the image to get the answer. You can't just guess based on text.
• The "Crop and Search" Strategy: Instead of showing the AI the whole messy photo, the test forces the AI to act like a real human researcher.
  • Analogy: Imagine you are looking at a crowded street scene. You can't just say "I see a person." You have to say, "I see a red umbrella in the bottom left corner. Let me zoom in on that umbrella, take a picture of just the logo, and search for that logo."
  • The new test forces the AI to do this multi-round cropping: Zoom in, search, zoom in again, search again, until it finds the truth.

3. The Results: Who Passed the Test?

The authors tested several top AI models (like Gemini, GPT-5, and open-source models) on this new, hard test.

• The "Lazy" Smart AIs: Some very powerful models (like Gemini) actually did worse when they were allowed to use search tools. Why? Because they were so confident in their own memory that they didn't bother to look at the clues. They tried to guess and failed.
• The "Hard-Working" AIs: Some slightly smaller models did surprisingly well. They didn't rely on guessing; they actually used the "crop and search" method to find the answers.
• The Magic Trick (MVF): The authors found that if they forced the AI to zoom in and search multiple times (a strategy they call "Multi-turn Visual Forcing"), the models got much better. It's like telling a detective: "Don't just look at the whole room. Go check the window, then the table, then the floor." This simple instruction made the AI much smarter at solving visual mysteries.

The Big Takeaway

The paper tells us that to build truly smart AI researchers, we can't just make the AI "smarter" with more data. We have to teach it how to look.

We need to stop giving AI easy tests where they can guess the answer, and start giving them messy, real-world puzzles where they have to zoom in, search, and connect the dots—just like a human would.

In short: The old tests let the AI cheat. The new test (VDR-Bench) forces the AI to do the hard work of looking, searching, and thinking, proving that the best AI researchers are the ones who know how to use their eyes, not just their memory.

Technical Summary

1. Problem Statement

The paper addresses the critical gap in evaluating Multimodal Large Language Models (MLLMs) within Vision-DeepResearch systems. While MLLMs have advanced in Visual Question Answering (VQA), existing benchmarks for multimodal search and deep research suffer from two fundamental flaws that prevent accurate assessment of real-world capabilities:

1. Lack of Visual-Search-Centricity: Many benchmark questions can be solved without genuine visual verification. Answers are often inferred through:
   • Cross-textual cues: Intersecting textual constraints in the question to guess the answer.
   • Prior World Knowledge: Relying on the model's pre-trained parametric knowledge rather than retrieving new visual evidence.
   • Consequence: High benchmark scores often reflect text retrieval or memorization rather than visual grounding.
2. Overly Idealized Evaluation Scenarios:
   • Image Search: Current benchmarks rely on "perfect retrieval," where querying the full original image yields an exact or near-duplicate match with metadata, bypassing the need for entity localization or iterative refinement.
   • Text Search: Questions are often shallow and direct, failing to stress-test multi-hop reasoning and evidence aggregation capabilities.
   • Consequence: These setups do not reflect the noisy, ambiguous, and iterative nature of real-world visual search, where relevant information is often fragmented across different crops, viewpoints, and contexts.

2. Methodology: VDR-Bench

To address these limitations, the authors introduce VDR-Bench (Vision-DeepResearch Benchmark), a large-scale dataset comprising 2,000 carefully curated VQA instances.

Data Curation Pipeline

The dataset is constructed via a rigorous, multi-stage, vision-centric pipeline designed to enforce visual necessity and prevent shortcuts:

1. Multi-Domain Pre-Filtering: Images are selected from diverse domains (Sports, Nature, Architecture, etc.) and filtered for quality and entity density.
2. Manual Cropping & Visual Search: Annotators manually crop salient regions (objects, logos, landmarks) rather than using the full image. These crops are used as queries for web-scale image search to generate candidate results.
3. Entity Extraction & Verification: Candidate entities are extracted from search results and verified by both MLLMs and human experts to ensure they cannot be trivially identified via full-image search.
4. Seed VQA Generation: Seed questions are generated requiring explicit recognition of the verified visual entity.
5. Knowledge-Graph Complexity Expansion: To ensure multi-hop reasoning, seed entities are linked to external knowledge graphs. Questions are expanded to require traversing nodes (e.g., "What is the headquarters city of the company whose logo is in the image?").
6. Solvability & Quality Verification: A two-stage check ensures answers can only be recovered by combining the visual search trajectory (cropped regions) with knowledge expansion, eliminating text-only shortcuts.

Proposed Workflow: Multi-Round Cropped-Search

Recognizing that current MLLMs struggle with visual retrieval, the authors propose a Multi-Round Cropped-Search workflow (specifically Multi-turn Visual Forcing, MVF):

• Instead of searching the whole image, the model iteratively identifies regions of interest, crops them, and performs targeted searches.
• This strategy refines visual queries, reduces background noise, and enables precise entity localization.

Evaluation Metrics

• Answer Accuracy: Measures the correctness of the final answer.
• Entity Recall (ER): A novel metric evaluating whether the system successfully discovers task-relevant entities during the search trajectory. It uses an LLM-as-a-judge to assess semantic equivalence between retrieved entities and gold-standard entity sequences, allowing for synonym recognition and multiple valid reasoning paths.

3. Key Contributions

1. Critical Analysis of Existing Benchmarks: The paper provides a quantitative analysis demonstrating that current benchmarks (e.g., SimpleVQA, LiveVQA, FVQA) are flawed due to text-leakage and perfect-retrieval bias, often overestimating model capabilities.
2. VDR-Bench: The introduction of a new, rigorous benchmark with 2,000 instances that enforces visual-search-centric reasoning, multi-hop complexity, and realistic search conditions.
3. Multi-Round Cropped-Search Strategy: The proposal and validation of a simple yet effective workflow (MVF) that significantly improves performance in realistic visual retrieval scenarios by forcing iterative localization.

4. Experimental Results

The authors evaluated state-of-the-art models (Gemini 2.5 Pro, GPT-5, Claude-4-Sonnet, Qwen3-VL variants) under three settings:

• Direct Answer: No search tools.
• CIS+TS: Cropped-Image Search + Text Search.
• CIS+TS+MVF: Adding Multi-turn Visual Forcing (iterative cropping).

Key Findings:

• Low Baseline Performance: Direct Answer scores were extremely low (e.g., Qwen3-30B scored ~3.8%), confirming that VDR-Bench cannot be solved by prior knowledge alone.
• The "Lazy Search" Phenomenon: Models with strong prior knowledge (e.g., Gemini 2.5 Pro, GPT-5) often failed to utilize search tools effectively when equipped with them, relying instead on internal knowledge. Conversely, models with weaker priors but better search integration (e.g., Qwen3-235B) showed strong search capabilities.
• Impact of MVF: The Multi-turn Visual Forcing strategy significantly boosted performance across all models.
  • Example: Gemini 2.5 Pro improved from 16.2% (CIS+TS) to 30.0% (CIS+TS+MVF).
  • Example: Qwen3-235B improved from 21.2% to 27.4%.
• Correlation: There is a strong positive correlation between Entity Recall and Answer Accuracy, proving that successful deep research depends on accurately locating and verifying visual entities first.

5. Significance

• Redefining Evaluation: VDR-Bench shifts the paradigm from "can the model guess?" to "can the model find and verify?" by eliminating shortcuts and perfect-retrieval biases.
• Practical Guidance: The results suggest that building robust Vision-DeepResearch agents requires more than just scaling base models; it necessitates architectural strategies that encourage iterative visual grounding and active tool use (like multi-round cropping).
• Real-World Applicability: By simulating the noisy, iterative nature of real-world search, VDR-Bench offers a more faithful testbed for developing agents capable of solving complex, real-world visual-textual problems.

The code and benchmark are released at: https://github.com/Osilly/Vision-DeepResearch.