🤖 AI

SearchEyes: Towards Frontier Multimodal Deep Search Intelligence via Search World Simulation

SearchEyes introduces a unified simulated search world built on a typed knowledge graph that integrates data, environment, and reward signals through Perception-Knowledge Chains and Hop-Anchored Policy Optimization, enabling state-of-the-art multi-hop reasoning performance in multimodal search agents without relying on external engines or separate reward models.

Original authors: Zhengbo Jiao, Yiming Cheng, Yilei Jiang, Kaituo Feng, Rui Huang, Tianyi Jiang, Juanxi Tian, Jiapeng li, Qunzhong Wang, Tailai Chen, Qianshan Wei, Chuan Xiao, Shanyu Rong, Yangfu Li, Yanhan Zhou, Yunpu

Published 2026-07-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhengbo Jiao, Yiming Cheng, Yilei Jiang, Kaituo Feng, Rui Huang, Tianyi Jiang, Juanxi Tian, Jiapeng li, Qunzhong Wang, Tailai Chen, Qianshan Wei, Chuan Xiao, Shanyu Rong, Yangfu Li, Yanhan Zhou, Yunpu Ma, Yifan Zhang, Xiangyu Yue

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart robot how to solve a complex mystery. The robot needs to look at a picture, read some documents, and connect the dots to find an answer. This is what the paper calls a "multimodal search agent."

However, the authors of this paper, SearchEyes, found that the way we usually teach these robots is broken. It's like trying to teach someone to drive by giving them a map, a car, and a driving test, but making sure none of them match.

The Map (Training Data): You give them a list of questions and answers, but you throw away the "how-to" steps.
The Car (Environment): You let them drive on a real, unpredictable public road (the actual internet) where the traffic lights might change or the road might disappear.
The Test (Rewards): You only tell them "Good job" or "Bad job" at the very end. If they made a mistake in step 1, but got lucky in step 10, they still get a "Good job." They never learn where they went wrong.

SearchEyes fixes this by building a perfect, self-contained simulation where the map, the car, and the test are all built from the same blueprint.

Here is how they did it, using simple analogies:

1. The Blueprint: A "Typed" Knowledge Graph

Instead of using the messy, real internet, the team built a giant, organized library of facts called a Knowledge Graph. Think of this as a massive family tree of the world's information, but with strict rules:

Every person (entity) has a photo, a biography, and a list of relationships.
They only keep the "famous" people who have all three (photo, bio, and connections) so the robot can practice looking at pictures and reading text.

2. The Training Course: "Perception-Knowledge Chains" (PKC)

To train the robot, they didn't just write random questions. They used a special recipe called Perception-Knowledge Chains (PKC).

The Analogy: Imagine a scavenger hunt where you must alternate between looking at a photo and reading a clue.
- Step 1 (Perception): "Look at this picture of a man. Who is he?" (The robot must identify the person from the image).
- Step 2 (Knowledge): "Now, look up what team he played for." (The robot must search text).
- Step 3 (Perception): "Find a photo of the stadium where that team plays."
The Magic: The system generates these questions automatically but keeps a "cheat sheet" (the exact path of facts) hidden in the background. This ensures the robot must take the long, multi-step route to solve the puzzle, rather than guessing the answer immediately.

3. The Driving Simulator: A "Self-Contained Search World"

In the real world, if you ask a search engine for something, the results might change tomorrow. In the SearchEyes world, the "search engine" is actually just a giant, pre-loaded database of the library they built.

The Benefit: It's 100% reproducible. If the robot searches for "Cristiano Ronaldo," it always gets the exact same top 5 results. This removes the chaos of the real internet, allowing the robot to learn the logic of searching without getting distracted by broken links or changing websites.

4. The Coach: "Hop-Anchored" Feedback (HaPO)

This is the most important innovation. In normal training, the coach only says "You got the answer right!" at the very end.

The Problem: If the robot took 10 steps to get there, the coach doesn't know which step was brilliant and which was a lucky guess.
The SearchEyes Solution: Because they kept the "cheat sheet" (the exact path of facts), they can act like a coach who watches every single step.
- "Great job finding the stadium photo in Step 3!"
- "Oops, you picked the wrong team in Step 2. Let's try that again."
The Metaphor: Instead of grading the whole exam with one score, they give a grade for every single question on the test. This helps the robot learn much faster because it knows exactly where to improve.

The Results

When they tested this new method (called SearchEyes) on six different difficult tests involving pictures and text:

It beat all other open-source robots that try to do the same thing.
Their smaller robot (27 billion "brain cells") performed better than much larger, expensive robots from big tech companies.
It proved that by giving the robot a structured, step-by-step training world and precise feedback, you don't need a massive supercomputer to get great results.

In short: SearchEyes stopped trying to teach robots by throwing them into the chaotic real internet. Instead, they built a perfect, controlled training gym where the robot practices specific skills, gets graded on every move, and learns to solve complex visual mysteries with incredible efficiency.

Technical Summary: SearchEyes

Problem Statement

Training multimodal search agents capable of long-horizon, multi-hop reasoning faces a fundamental structural disconnect in current pipelines. Existing approaches construct training data, search environments, and reward signals independently, leading to three critical issues:

Data-Environment Decoupling: Training data synthesis often discards intermediate structural metadata (e.g., path structure, entity IDs), while search environments rely on irreproducible external engines with non-deterministic APIs.
Reward Sparsity: Reinforcement learning (RL) signals are typically computed only at the trajectory level (based on the final answer). For multi-hop reasoning, the probability of a correct final answer diminishes sharply with the number of hops, rendering trajectory-level rewards too sparse for effective policy optimization.
Credit Assignment Failure: Without step-level signals, agents struggle to learn which specific actions in a long chain contributed to success or failure, especially in complex multimodal tasks requiring both visual perception and textual knowledge retrieval.

Methodology: SearchEyes

SearchEyes addresses these bottlenecks by unifying data synthesis, environment simulation, and RL training through a simulated search world built on a typed knowledge graph (Wikidata5M). The framework consists of three tightly coupled components:

1. Unified Search World Simulation

The system constructs a self-contained search environment using a typed knowledge graph $G = (V, R, T, \phi)$ derived from Wikidata5M, Wiki6M (text), and Wikipedia images.

Self-Contained Retrieval: Instead of external APIs, the environment uses a hybrid retrieval engine (Dense embeddings + BM25) fused via Reciprocal Rank Fusion (RRF) over a pre-indexed document collection. This ensures deterministic, reproducible, and auditable retrieval.
Tool Set: Agents interact via deterministic tools: text_search, visual_search (image cropping + embedding retrieval), lookup, summarize, and python_interpreter.

2. Perception-Knowledge Chain (PKC) Synthesis

To generate high-quality training data, SearchEyes introduces a pipeline that samples constrained multi-hop paths over the intersection of visual and knowledge entities:

Constrained Path Sampling: Paths are sampled via constrained random walks with strict constraints:
- P-K Alternation: Enforcing alternation between Perception hops (identifying visual entities from images) and Knowledge hops (text-based search).
- Disambiguating Constraints: Introducing "one-to-few" predicates and constraint edges to prevent trivial single-path solutions, increasing the reasoning graph's treewidth.
- Domain Diversity: Ensuring paths traverse multiple semantic domains (Person, Work, Org, Geo).
- Anti-Shortcut Filtering: Excluding hub entities and meta-level predicates to avoid guessable paths.
Information Concealment: Questions are generated such that entity names and full relation sequences are hidden. The anchor entity is presented only via an image, and intermediate relations are obfuscated to domain-level hints, forcing the agent to perform genuine multi-step reasoning.
SFT Data Distillation: Expert trajectories are synthesized using a "privileged" environment with retrieval boosts and observation denoising. These trajectories are then exported with raw, noisy observations for Supervised Fine-Tuning (SFT), teaching the model to reason over realistic retrieval results.

3. Hop-Anchored Policy Optimization (HaPO)

To solve the reward sparsity problem, SearchEyes proposes HaPO, a post-training algorithm that reuses the gold entity sequence from PKC synthesis for step-level credit assignment without a separate process reward model.

Hop-Anchored Groups: Trajectories are grouped by the intermediate gold entities they successfully retrieve. Within these groups, advantages are computed relative to the group's outcome, isolating the effect of actions taken after reaching a specific intermediate state.
Combined Advantage: The final advantage signal interpolates between trajectory-level and hop-anchored step-level advantages.
Fatal-Aware Masking: The algorithm detects and masks degenerate trajectory suffixes caused by consecutive tool errors, preventing gradient noise from corrupting policy updates.
Smooth Asymmetric Gating: Replaces hard clipping with a sigmoid-based gate to stabilize long-horizon optimization, applying stronger damping to negative advantages.

Key Contributions

Unified Search World Simulation: A novel framework using a typed knowledge graph to co-design data synthesis, environment simulation, and RL training, eliminating the data-environment-reward disconnect.
Perception-Knowledge Chain (PKC) Synthesis: A KG-driven pipeline that generates constrained, multi-hop visual-language questions with strict structural guarantees (P-K alternation, disambiguation) and information concealment.
Agentic Post-Training Pipeline (HaPO): A training algorithm that leverages retained structural metadata (gold entity IDs) to enable step-level credit assignment, blending hop-anchored and trajectory-level signals without requiring a trained process reward model.
VisSearch Bench: A new benchmark introduced for evaluating multi-hop visual search with P-K alternating structures.

Experimental Results

SearchEyes was evaluated on six multimodal knowledge-intensive benchmarks (SimpleVQA, VDR, MMSearch, LiveVQA, BC-VL, FVQA) and the new VisSearch Bench.

Performance: SearchEyes-27B achieved state-of-the-art performance among open-source multimodal search agents, improving the strongest open-source baseline by 6.2 points on average.
Parameter Efficiency: SearchEyes-9B matched the performance of 30B-scale baselines (e.g., OpenSearch-VL-30B) while using 3.3x fewer parameters, demonstrating superior data efficiency.
Ablation Studies:
- Removing PKC constraints (e.g., P-K alternation, anti-shortcut filtering) caused significant performance drops (up to -7.7 average points), confirming the necessity of structural constraints for learning genuine multi-step reasoning.
- Replacing HaPO with standard GRPO resulted in a -4.0 average point drop, validating that step-level credit assignment is the primary bottleneck in multi-hop search.

Significance

The paper argues that the primary limitation in current multimodal search agents is not model capacity, but the structural disconnect between data, environment, and reward. By treating the knowledge graph as a unified backbone, SearchEyes demonstrates that data-environment-algorithm co-design is essential for training effective long-horizon agents. The results suggest that retaining structural metadata throughout the training pipeline allows for precise step-level credit assignment, enabling smaller models to achieve performance competitive with much larger proprietary systems. The work provides a reproducible, open-source recipe for building strong multimodal search agents without reliance on external, non-deterministic APIs.