💻 computer science

From Web to Pixels: Bringing Agentic Search into Visual Perception

This paper introduces "Perception Deep Research," a new open-world visual perception challenge where objects must be identified via external knowledge before localization, and addresses it with the WebEye benchmark and the Pixel-Searcher agentic framework, which demonstrates state-of-the-art open-source performance in search-based grounding, segmentation, and VQA.

Original authors: Bokang Yang, Xinyi Sun, Kaituo Feng, Xingping Dong, Dongming Wu, Xiangyu Yue

Published 2026-05-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bokang Yang, Xinyi Sun, Kaituo Feng, Xingping Dong, Dongming Wu, Xiangyu Yue

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are playing a game of "Where's Waldo?" (or "Where's Wally?"), but with a twist.

In the classic game, you look at a crowded picture and find Waldo based on what he looks like: a red-and-white striped shirt, a hat, and glasses. You don't need to know who Waldo is; you just need to spot the visual pattern.

This paper argues that most current AI vision systems are stuck playing the classic game. They are great at finding things that look like a "red car" or a "person in a blue shirt." But they struggle when the question isn't about what something looks like, but about who it is based on facts you can't see in the picture.

The New Challenge: "The Mystery Guest"

The authors introduce a new, harder version of the game called Perception Deep Research.

Imagine a photo of a crowded party. The question isn't "Find the person in the red shirt." Instead, the question is:

"Find the person who became the global ambassador for a skincare brand in January 2026, after that brand was bought by a company for $2.7 billion."

You cannot solve this just by looking at the photo. The photo doesn't show the price tag of the acquisition or the date of the ambassador announcement. To find the right person, the AI must:

Go on a Web Hunt: Search the internet to figure out which brand was bought, which month, and who the ambassador was.
Solve the Mystery: Realize, "Ah, the person in the photo named 'Winter' is the one we are looking for."
Point to the Pixel: Finally, go back to the photo and draw a box around Winter.

The paper calls this "From Web to Pixels." It's about connecting internet facts to specific spots in an image.

The Tools They Built

To test if AI can do this, the team built two main things:

1. WebEyes (The Test Course)
Think of this as a specialized obstacle course for AI. It contains 120 complex images and hundreds of tricky questions.

The Twist: Every question requires the AI to look up hidden facts (like a celebrity's recent role, a product's launch date, or a character's backstory) before it can even guess where to look in the image.
The Goal: The AI must not just answer the question; it must point to the exact person or object in the picture that matches the answer.

2. Pixel-Searcher (The Smart Detective)
This is the new AI agent the authors created to take the test. Instead of just staring at the image and guessing, Pixel-Searcher acts like a detective with a magnifying glass and a laptop.

Step 1: The Search. It breaks the question down. "Who was the ambassador?" "When was the deal?" It searches the web and reads the results.
Step 2: The Deduction. It connects the dots. "Okay, the deal happened in January, and the ambassador is Winter."
Step 3: The Hunt. It goes back to the image. It looks for a person who looks like Winter (maybe checking for specific clothes or features mentioned in the search) and draws a box around them.

What They Found

The researchers tested Pixel-Searcher against other smart AI models. Here is the verdict:

Old AI Models: When faced with these "Mystery Guest" questions, standard AI models got confused. They tried to guess based only on what they saw in the picture or what they already "knew" from their training. They often picked the wrong person because they couldn't find the hidden facts.
Pixel-Searcher: This new agent performed the best among open-source models. By actually searching the web and reasoning through the clues, it successfully found the right targets much more often.

Where it still struggles:
The paper notes that the AI doesn't fail because it can't draw a perfect box around a person. It fails earlier in the process:

Bad Search: It sometimes searches for the wrong thing or misses a key fact.
Wrong Identity: It figures out the right facts but picks the wrong person in the photo (e.g., it knows the ambassador is "Winter," but it points to the wrong person named Winter).
Binding: It struggles to connect the "fact" (Winter) to the "visual" (the person in the photo).

The Big Picture

The paper concludes that for AI to truly understand the world, it can't just be a "camera" that sees shapes. It needs to be a "researcher" that can look up information, solve puzzles, and then use that knowledge to point at the right thing in a picture. They call this new field Perception Deep Research, and they've provided the tools (WebEyes and Pixel-Searcher) to help other scientists build better "detective" AIs.

Technical Summary: From Web to Pixels: Bringing Agentic Search into Visual Perception

1. Problem Definition: Perception Deep Research

The paper identifies a critical gap in current visual perception systems. While existing multimodal large language models (MLLMs) have advanced from recognizing visible categories to grounding implicit targets based on internal knowledge, they struggle in open-world settings. In these scenarios, the target object is visible in an image, but the decisive evidence required to identify it lies outside the image and beyond the model's frozen parametric knowledge.

The authors formalize this challenge as Perception Deep Research. Unlike standard referring expression comprehension or reasoning segmentation where clues are visual or internal, this setting requires a model to:

Resolve a target identity from external facts (e.g., recent events, long-tail entities, multi-hop relations, creator affiliations).
Bind this resolved identity to a concrete visual instance (box, mask, or region-level answer) within the image.

The core difficulty lies in the coupling of two steps: transforming indirect factual clues into an explicit target hypothesis, and then verifying that this hypothesis is compatible with the observed visual region, distinguishing it from distractors or visually similar objects.

2. Methodology

2.1 The WebEyes Benchmark

To measure progress in this domain, the authors introduce WebEyes, an object-anchored benchmark designed with verifiable evidence and precise annotations.

Construction: The dataset follows an "object-first" workflow. Annotators select multi-instance images, mark foreground objects, and then use an agent to perform chained web searches to gather evidence (roles, release dates, brand affiliations) before generating knowledge-intensive queries.
Scale: It comprises 120 images, 473 annotated object instances, and 645 unique QA pairs, resulting in 1,927 task samples.
Task Views: WebEyes evaluates three complementary tasks:
1. Search-based Grounding: Predicting a bounding box from a query requiring external knowledge.
2. Search-based Segmentation: Predicting a pixel-level mask.
3. Search-based VQA: Selecting the correct knowledge-rich description for a highlighted region.
Quality Control: The pipeline includes automated filtering to remove "shortcuts" (questions solvable by closed-book knowledge or vision alone) and manual verification to ensure evidence correctness and target uniqueness.

2.2 Pixel-Searcher: An Agentic Search-to-Pixel Workflow

The authors propose Pixel-Searcher, a reference workflow that decomposes the task into an agentic loop rather than a direct end-to-end prediction. It operates in two phases:

Agentic Search & Target Resolution:
- The agent analyzes the query and decomposes it into searchable sub-goals.
- It enters a Search-Reason loop, alternating between retrieving external evidence (via search tools), reasoning over the accumulated facts, and resolving a structured hypothesis $h = \{e, c, K\}$ , where $e$ is the entity name, $c$ is the visual category, and $K$ represents image-checkable cues.
- The agent verifies that the resolved entity is not an intermediate clue but the final visible target.
Agentic Grounding & Tool Use:
- Using the resolved hypothesis $h$ , the agent guides visual grounding tools.
- For Grounding/Segmentation: It identifies candidate regions, verifies them against the resolved evidence, and invokes a promptable segmentation tool (SAM3) for mask refinement.
- For VQA: It reverses the process, resolving each answer option into evidence-aware cues and selecting the one that best matches the grounded visual region.

3. Key Contributions

The paper makes three primary contributions:

Perception Deep Research: A new formulation for open-world visual perception where models must actively seek external evidence to resolve hidden identities before grounding them.
WebEyes Benchmark: A dataset featuring verifiable external evidence, knowledge-intensive queries, and precise annotations across grounding, segmentation, and VQA views, specifically designed to test the ability to convert web knowledge into pixel-level outputs.
Pixel-Searcher: An agentic workflow that demonstrates the efficacy of search-to-pixel reasoning. The authors provide diagnostic experiments revealing that performance bottlenecks lie primarily in evidence acquisition, identity resolution, and visual instance binding, rather than in the final mask refinement step.

4. Experimental Results

Experiments were conducted on WebEyes using various open-source and closed-source models without task-specific fine-tuning.

Performance: Pixel-Searcher achieves the strongest performance among all open-source methods across all three task views.
- In Search-based Grounding, it improves the IoU of the baseline Qwen3-VL-8B from 26.81% to 34.17% and Recall@0.5 from 32.61% to 41.30%.
- In Search-based Segmentation, it improves gIoU from 35.78% to 39.17% and cIoU from 25.94% to 32.41%.
- In Search-based VQA, it achieves 42.24% accuracy, outperforming other open-source models.
Comparison: While Pixel-Searcher leads open-source methods, a performance gap remains compared to top-tier closed-source models (e.g., Doubao-Seed, Gemini), suggesting that evidence selection and entity resolution remain challenging even for advanced agents.
Ablation & Failure Analysis:
- Removing direct localization cues causes the most significant performance drops.
- Failure Modes: Approximately 78% of failures (304 out of 389) occur in the search/entity resolution phase (incorrect entity identification or search planning). Only a small fraction (10 cases) are due to box-to-mask transfer errors. This confirms that the primary bottleneck is identifying the correct entity rather than refining the visual boundary.

5. Significance and Claims

The paper claims that WebEyes provides the necessary infrastructure to benchmark and advance "Perception Deep Research," moving visual perception beyond static image analysis to dynamic, knowledge-driven interaction.

The significance of Pixel-Searcher lies in demonstrating that agentic search is a viable and necessary mechanism for open-world visual perception. The authors argue that for tasks where the target identity is hidden behind external facts, simply relying on visual attributes or frozen model knowledge is insufficient. The workflow proves that resolving the "who" (identity) before determining the "where" (location) is critical, and that failures in this domain are predominantly semantic (search/reasoning errors) rather than geometric (mask refinement errors).

The paper concludes that this direction requires models to actively seek external evidence and verify identities against visual instances, offering a simple but effective starting point for studying how agentic search can bridge the gap between web knowledge and pixel-level perception.