🤖 machine learning

HyperEyes: Dual-Grained Efficiency-Aware Reinforcement Learning for Parallel Multimodal Search Agents

HyperEyes is a parallel multimodal search agent trained via a dual-grained efficiency-aware reinforcement learning framework that fuses visual grounding and retrieval into concurrent actions, achieving superior accuracy and significantly reduced inference costs compared to existing sequential agents.

Original authors: Guankai Li, Jiabin Chen, Yi Xu, Xichen Zhang, Yuan Lu

Published 2026-05-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Guankai Li, Jiabin Chen, Yi Xu, Xichen Zhang, Yuan Lu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery involving six different suspects in a single photo.

The Old Way (Serial Search):
Traditional AI agents work like a very thorough but slow detective. They look at the photo, pick the first person, crop their face, search the internet for that person, read the results, and then move to the second person. They repeat this process one by one.

The Problem: If the photo has six people, the detective makes six separate trips to the library. By the time they get to the sixth person, the first five trips have cluttered their notebook with too much information, making it hard to focus. It's like trying to cook a six-course meal by boiling one pot of water, waiting for it to finish, then boiling the next pot. It takes forever and wastes energy.

The New Way (HyperEyes):
The paper introduces HyperEyes, a new AI agent that follows the philosophy: "Search wider, not longer."
Instead of visiting the library six times, HyperEyes looks at the whole photo, identifies all six suspects at once, and sends six search requests simultaneously in a single trip. It's like sending six different researchers to the library at the same time, all working in parallel, and bringing back a single, organized report.

How HyperEyes Was Trained (The "School" Analogy)

The researchers didn't just tell the AI to be faster; they built a special training school for it with two main phases:

1. The "Strict Teacher" Phase (Data Synthesis):
First, they created a massive library of practice problems. But they didn't just want any correct answer; they wanted the fastest correct answer.

They used a technique called Progressive Rejection Sampling. Imagine a teacher giving a student a test. If the student takes 10 minutes to solve a problem, the teacher throws that attempt away. If the student solves it in 2 minutes, the teacher keeps it. They kept only the fastest, most efficient solutions to teach the AI how to be snappy from day one.

2. The "Double-Grained" Coaching Phase (Reinforcement Learning):
Once the AI knew the basics, they applied a special coaching system with two levels:

Macro Level (The Big Picture): They gave the AI a reward system called TRACE. Think of this as a coach who says, "If you solve the mystery in 3 steps, you get a gold star. If you take 5 steps, you get a penalty." Crucially, the coach gets stricter over time. If the AI learns to solve it in 3 steps, the coach raises the bar to 2 steps. This forces the AI to constantly get more efficient, not just accurate.
Micro Level (The Fine Details): Sometimes the AI makes a mistake halfway through. The On-Policy Distillation (OPD) acts like a "ghostwriter" who steps in only when the AI fails. The ghostwriter (a smarter, larger AI) whispers the correct next word to the student AI, teaching it exactly where it went wrong without wasting time on the parts it already got right.

The New Scorecard (IMEB)

The researchers realized that old tests only cared if the AI got the answer right, ignoring how long it took or how many "library trips" (tool calls) it made.
They created a new benchmark called IMEB (Image Multi-Entity Benchmark).

The Analogy: Imagine grading a race. Old tests only looked at who crossed the finish line first. The new test (IMEB) looks at who crossed the finish line and who used the least amount of fuel. They introduced a Cost-Aware Score that rewards accuracy but heavily penalizes wasting time and energy.

The Results

When they put HyperEyes to the test against the best existing AI agents:

Accuracy: It got the right answer more often than its competitors (specifically, 9.9% better than the best open-source rival).
Efficiency: It was dramatically faster. It needed 5.3 times fewer "library trips" to solve the same problems.

Summary

HyperEyes is like upgrading from a single-lane road with traffic jams to a multi-lane highway. By teaching the AI to look at all the clues at once and punishing it for taking unnecessary detours, the researchers created an agent that is not only smarter but also significantly more efficient, solving complex visual puzzles in a fraction of the time.

Technical Summary: HyperEyes

Problem Statement

Existing multimodal search agents predominantly rely on a sequential paradigm, processing target entities one by one. While effective for multi-hop reasoning, this approach incurs severe interaction redundancy when queries naturally decompose into independent sub-retrievals (e.g., identifying multiple entities in a single image). Conventional agents suffer from "crop-then-search" pipelines where early localization errors corrupt downstream results, and the sequential nature forces multi-entity queries into verbose, redundant interaction rounds. Furthermore, current training paradigms optimize primarily for final answer accuracy, lacking incentives to prefer compact parallel trajectories over verbose ones. Consequently, parallel capabilities often degrade into brute-force over-searching, inflating latency and token consumption without improving accuracy.

Methodology

The authors propose HyperEyes, a parallel multimodal search agent designed around the principle of "search wider, not longer." The system fuses visual grounding and retrieval into a single atomic action, enabling concurrent search across multiple entities within a single turn.

1. Unified Grounded Search (UGS)

HyperEyes reformulates visual grounding from a prerequisite step into a parameter of the retrieval action. Instead of a two-stage process (crop then search), the agent simultaneously predicts bounding boxes for all target entities and dispatches parallel search queries (both image and text) within a single turn. This decouples entity identification from downstream reasoning.

2. Training Pipeline

The training process consists of two stages:

Stage 1: Parallel-Amenable Data Synthesis & Cold-Start Supervision
- Data Synthesis: The authors curate a dataset of 271,000 tool-dependent tasks, including 20,000 visual multi-entity queries (synthesized via mosaic augmentation of fine-grained classification datasets) and 5,000 textual multi-constraint queries.
- Progressive Rejection Sampling (PRS): To generate high-quality Supervised Fine-Tuning (SFT) trajectories, the authors employ PRS. This algorithm samples trajectories across ascending turn budgets, strictly retaining only the shortest successful trajectory for each query. This filters out redundant tool calls and iterative reformulations, distilling 30,000 efficiency-oriented cold-start trajectories.
Stage 2: Dual-Grained Efficiency-Aware Reinforcement Learning (RL)
To optimize for both accuracy and efficiency, the authors introduce a two-level RL framework:
- Macro Level (TRACE): A trajectory-level reward mechanism called Tool-use Reference-Adaptive Cost Efficiency (TRACE). It utilizes a reference threshold for tool-call rounds ( $\hat{t}_c$ ) that monotonically tightens during training based on the best-performing rollouts of the current epoch. This creates an implicit curriculum that suppresses superfluous tool calls without over-restricting genuine multi-hop searches. The reward penalizes trajectories exceeding the reference or failing to meet accuracy, while rewarding those within the efficient bounds.
- Micro Level (OPD): To address the credit-assignment deficiency of sparse outcome rewards on failed rollouts, the authors adapt On-Policy Distillation (OPD). A stronger "teacher" model (HyperEyes-235B) provides dense, token-level corrective signals exclusively to the "student" model (HyperEyes-30B) on failed trajectories. This allows the student to learn from valid intermediate reasoning steps without inheriting the teacher's inference cost.

3. Evaluation Benchmark (IMEB)

Recognizing that existing benchmarks evaluate accuracy while ignoring efficiency, the authors introduce IMEB (Image Multi-Entity Benchmark). This human-curated dataset comprises 300 instances requiring concurrent localization and retrieval across multiple entities. They also propose the Cost-Aware Score (CAS), a metric that jointly quantifies reasoning correctness and search efficiency by penalizing token consumption and tool-call rounds.

Key Contributions

HyperEyes Agent: A parallel multimodal search agent operating on a Unified Grounded Search action space, achieving high efficiency by grounding and searching multiple entities concurrently.
Dual-Grained RL Framework: A novel training approach combining macro-level adaptive efficiency constraints (TRACE) with micro-level token-level distillation (OPD) to optimize search behavior.
Efficiency-Aware Benchmark (IMEB): The first benchmark to jointly evaluate answer accuracy and search efficiency in multi-entity visual scenarios, accompanied by the CAS metric.
Data Synthesis Pipeline: A method for generating parallel-amenable training data using Progressive Rejection Sampling to distill zero-redundancy trajectories.

Experimental Results

HyperEyes was evaluated on six multimodal search benchmarks (MMSearch, FVQA, LiveVQA, BCVL, MMSearch+, and IMEB) using Qwen3-VL backbones (30B and 235B).

Accuracy and Efficiency: HyperEyes-30B (RL) surpasses the strongest comparable-scale open-source agent (VDR) by 9.9% in accuracy while requiring 5.3× fewer tool-call rounds on average.
State-of-the-Art Performance: HyperEyes-235B (RL) approaches the performance of the proprietary Gemini-3.1-Pro, outperforming other open-source and commercial agents (e.g., Claude-Opus-4.6, Kimi-K2.5) in the agentic workflow setting.
Cost-Aware Score: Under the CAS metric, HyperEyes-30B achieves a score 4.3× higher than the second-best open-source competitor on complex multi-hop benchmarks and 7.6× higher on the multi-entity IMEB benchmark.
Ablation Studies: Experiments confirm that rigorous quality filtering (reducing data volume by 75%) significantly improves accuracy over raw data volume. Furthermore, the adaptive TRACE reward and efficiency-aligned teacher for OPD are critical for reducing tool-call redundancy and boosting accuracy.

Significance and Claims

The paper claims that HyperEyes establishes a new standard for open-source search agents, demonstrating that accuracy and operational efficiency are complementary rather than conflicting objectives in agentic multimodal search. By shifting from sequential "search longer" to parallel "search wider," the framework eliminates the redundancy and noise accumulation inherent in conventional crop-then-search pipelines. The authors assert that their work provides a reproducible foundation for building efficient, grounded multimodal assistants, particularly in domains where verifiable, source-grounded answers are essential. The release of IMEB and the training pipeline aims to lower the entry barrier for research into efficiency-aware multimodal agents.