Technical Summary: InterLV-Search

Problem Statement

Existing benchmarks for multimodal agentic search primarily evaluate the acquisition of textual evidence or treat visual information as a static input or a final answer endpoint. In these frameworks, visual evidence is either confined to the initial user query or retrieved solely to answer a local visual question, after which the search trajectory terminates. This formulation overlooks a critical capability in realistic information seeking: visual evidence as a search pivot. In complex, open-world tasks, visual observations (e.g., logos, inscriptions, spatial relations, or specific visual entities) often contain fine-grained cues that disambiguate the current state and determine the next search target, query, or branching decision. Current systems struggle with interleaved multimodal search, where agents must dynamically switch between visual and textual evidence acquisition, using evidence from one modality to condition retrieval actions in the other.

Methodology

The InterLV-Search Benchmark

The authors introduce InterLV-Search, a benchmark comprising 2,061 examples designed to evaluate Interleaved Language-Vision Agentic Search. The benchmark is structured into three progressively challenging levels:

1. Level 1: Active Visual Evidence Seeking

   • Goal: Evaluate the primitive ability to seek visual evidence from textual information needs.
   • Task: Formulated as a "Search-to-VQA" task. The agent receives a textual description of a visual query about an implicitly specified target. It must infer the hidden entity, retrieve the corresponding image, and answer a fine-grained visual question (e.g., color, material, pattern) grounded in that image.
   • Construction: Fully automated pipeline using MMKG-W (a Wikimedia-based multimodal knowledge graph with ~15K entities). The process involves generating implicit target-search subqueries and corresponding VQA subqueries, composed into natural language questions.

2. Level 2: Controlled Offline Interleaved Multimodal Search

   • Goal: Evaluate multi-hop interleaved evidence search in a controlled environment, isolating the capability to use visual evidence as intermediate pivots.
   • Task: Requires at least two rounds of visual evidence retrieval. The agent must ground a visual clue, convert it into the next retrieval target, and finally ground a terminal image to answer.
   • Types:
     • Initial-visual-probed: Visual probing is injected at the start of the reasoning chain.
     • Intermediate-visual-probed: Visual grounding is required at a middle stage, where a "bridge" entity (visually similar to an intermediate node) redirects the search path.
   • Construction: Fully automated pipeline leveraging entity-relation annotations in MMKG-W to construct verifiable multi-hop paths and obfuscate search chains.

3. Level 3: Open-Web Interleaved Multimodal Search

   • Goal: Evaluate the same capabilities in a noisy, dynamic open-web setting.
   • Task: Agents operate over real webpages and search results. This level includes both standard single-chain examples and multi-branch examples, where agents must explore parallel reasoning routes, gather evidence, compare entities, and select a branch to continue.
   • Construction: A machine-led, human-supervised pipeline. GPT-5.4-Thinking generates seed questions and candidate chains, which are then verified and refined by PhD-level human annotators to ensure validity, answerability, and interleaved structure. A CP-SAT optimizer selects the final subset to balance difficulty and minimize "no-search" solvability.

InterLV-Agent Framework

To standardize evaluation, the authors provide InterLV-Agent, a reference framework featuring:

• Reason-Act-Observe Loop: Agents plan, invoke tools, and observe results.
• Tool Suite: Includes image search, reverse image search, web search, webpage browsing, image cropping, and code execution.
• Memory: A lightweight two-level memory system (short-term for recent interactions, long-term for summarized history) to maintain search state across long trajectories.
• Evaluation Protocol: Uses final-answer accuracy judged by semantic equivalence (allowing for aliases and paraphrases) with specific interaction budgets (3, 7, and 10 steps for Levels 1, 2, and 3 respectively).

Key Contributions

1. InterLV-Search Benchmark: The first benchmark to jointly cover text-to-visual search, visual multi-hop retrieval, recurrent vision–text interleaving, and multimodal multi-branch search. It shifts the paradigm from "visual evidence as an endpoint" to "visual evidence as a search pivot."
2. Scalable Data Construction: Automated pipelines for Levels 1 and 2, and a semi-automated, human-in-the-loop pipeline for Level 3, enabling the generation of high-quality, interleaved multimodal search data.
3. Comprehensive Evaluation: A standardized framework (InterLV-Agent) and extensive experiments on both proprietary and open-source multimodal agents.

Experimental Results

Experiments on proprietary models (e.g., GPT-5.4, Gemini-3.1-Pro, Claude-Sonnet-4.6) and open-source agents (e.g., MMSearch-R1, SenseNova-MARS-32B) reveal:

• Performance Gap: Current systems remain far from solving interleaved multimodal search. Even the best-performing model achieves below 50% overall accuracy.
• Necessity of Search: Tool-free performance is limited (best ~20% on Level 3), confirming that parametric knowledge alone is insufficient.
• Tool Use Variance: While proprietary models show consistent gains with tool use, open-source search-oriented agents often show negligible gains or performance degradation, indicating limitations in search planning and visual grounding.
• Difficulty Progression: Performance decreases from Level 1 to Level 3. Within Level 3, agents perform significantly worse on multi-branch tasks compared to single-chain tasks, highlighting struggles with non-linear search control.
• Retrieval Bottleneck: Decomposition analysis shows that while agents can answer correctly once the target entity is retrieved (high accuracy given retrieval), the retrieval recall (R@5) remains low, particularly in Level 2. Target evidence localization is a major bottleneck.
• Visual Pivots: Trajectory analysis confirms that successful agents frequently use visual evidence as search pivots, but many models fail to maintain this interleaved pattern effectively.

Significance

The paper claims that InterLV-Search exposes challenges not captured by existing benchmarks. The results demonstrate that success in open-world multimodal search depends not merely on accessing external tools, but on the ability to:

1. Locate intended visual evidence.
2. Use that evidence as a dynamic pivot to guide subsequent retrieval decisions.
3. Maintain a coherent search state across long or branching trajectories.

The substantial performance gaps observed suggest that current multimodal agents are not yet robust enough for complex, interleaved multimodal search, and the benchmark provides a necessary foundation for future work in agents that can reliably acquire, connect, and act on multimodal evidence during search.