🤖 AI

CVSearch: Empowering Multimodal LLMs with Cognitive Visual Search for High-Resolution Image Perception

CVSearch is a training-free, adaptive framework that enhances multimodal large language models' high-resolution image perception by dynamically combining expert-assisted search with a novel semantic-aware scanning mechanism to balance coverage and efficiency.

Original authors: Liupeng Li, Haoqian Kang, Zhenyu Lu, Jinpeng Wang, Bin Chen, Ke Chen, Yaowei Wang

Published 2026-05-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Liupeng Li, Haoqian Kang, Zhenyu Lu, Jinpeng Wang, Bin Chen, Ke Chen, Yaowei Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, high-resolution photograph of a busy city street, and you ask a computer, "Where is the tiny red car parked near the bakery?"

Current AI models (Multimodal Large Language Models) are like people with very strong brains but very weak eyes. They can understand language perfectly, but when they look at a massive image, they often squint it down to a small, blurry thumbnail to save time. In doing so, they miss the tiny red car entirely.

To fix this, researchers built CVSearch. Think of CVSearch not as a new brain, but as a smart pair of glasses and a search strategy that helps the AI look at the image the way a human detective would.

Here is how it works, broken down into simple steps:

1. The "Glimpse" (The Quick Check)

When you ask a question, CVSearch first takes a quick, casual look at the whole image.

The Analogy: Imagine walking into a room and asking, "Is there a cat here?" If you see a cat immediately on the sofa, you don't need to open every drawer. You just say, "Yes, there."
What the paper says: If the AI feels confident it has enough info from the whole picture, it answers immediately. This saves a huge amount of time.

2. The "Expert Assistant" (The Quick Scan)

If the AI isn't sure (maybe the object is small or hidden), it calls in a "Visual Expert" (a tool called SAM 3).

The Analogy: This is like calling a security guard who is great at spotting things. You say, "Look for a red car." The guard points to a spot.
The Problem: Sometimes, the guard misses it. Maybe the car is tiny, or the guard is just having a bad day. If the guard fails, the old methods would just give up and say, "I can't find it."
What the paper says: CVSearch doesn't give up. If the expert fails, it treats that failure as a signal to switch to a more thorough mode.

3. The "Smart Detective" (The Deep Dive)

This is the paper's biggest innovation. If the expert fails, CVSearch doesn't just scan the whole picture blindly. It uses a Cognitive Assess-then-Search workflow.

The "Smart Grid" (Semantic Guided Adaptive Patching):
- Old Way: Imagine trying to find a needle in a haystack by cutting the haystack into perfect, rigid square blocks. You might cut the needle in half, making it hard to recognize.
- CVSearch Way: Instead of rigid squares, CVSearch looks at the "shape" of the image. It cuts the image into pieces that follow the objects naturally. It keeps the whole car together in one piece and the sky in another. It doesn't slice the car in half.
- The Analogy: Instead of cutting a pizza into a grid, you cut it along the natural lines of the toppings so every slice has a whole pepperoni.
The "Bottom-Up" Search:
- Old Way: Most search methods start at the top (the big picture) and drill down. If they make a mistake at the top, they keep making mistakes all the way down.
- CVSearch Way: It starts at the bottom (the tiny, detailed pieces) and works its way up.
- The Analogy: Imagine trying to find a specific person in a crowd. Instead of looking at the whole crowd and guessing, you look at the faces of individual people first. Once you find a face that looks like the target, you zoom out to see where they are standing. This prevents the AI from getting lost in the big picture.

4. The "Focus Filter" (Visual Complexity)

CVSearch is also lazy in a smart way. It knows that the sky or a blank wall doesn't need much attention.

The Analogy: If you are looking for a specific person, you don't stare at the empty blue sky. You focus your energy on the busy part of the street where people are.
What the paper says: It calculates a "Visual Complexity Score." If a part of the image is boring (low complexity), it ignores it. If it's busy and detailed (high complexity), it spends more time looking there.

The Result

The paper claims that by combining these three things—checking if a quick look is enough, using an expert first, and then doing a smart, shape-aware deep dive if needed—CVSearch is much faster and more accurate than previous methods.

Old methods were either too slow (checking every single square inch) or too fragile (relying entirely on the expert who might miss things).
CVSearch is like a detective who knows when to take a quick glance, when to call a specialist, and when to use a magnifying glass on the most interesting parts of the crime scene, all while keeping the evidence (the objects) intact.

The paper demonstrates this on high-resolution images (like 8K photos) where finding tiny details is hard, showing that their method finds the "red car" and the "tiny helmet" much better than other AI systems without needing to retrain the AI's brain.

Technical Summary: CVSearch

Problem Statement

Multimodal Large Language Models (MLLMs) face a critical bottleneck in high-resolution (HR) image perception. Current models typically rely on fixed-resolution processing schemes that aggressively downsample input images, rendering them "blind" to small objects and fine-grained details essential for real-world tasks. While recent research has attempted to address this through cropping-based paradigms, HR visual encoders, and visual search, existing solutions struggle with a fundamental trade-off between coverage and efficiency:

Cropping-based methods preserve local details but sever spatial coherence, causing "semantic fragmentation" where objects spanning crop boundaries are split into disjoint tokens.
HR Visual Encoders inject high-frequency features but often lack flexibility in handling varying aspect ratios and still rely on global processing.
Visual Search Paradigms offer active perception but fall into two sub-categories with distinct limitations:
- Visual Expert-Assisted Search (e.g., SEAL, DyFo) is efficient but fragile; if the external expert fails to generate accurate proposals (common with tiny or occluded objects), the system suffers from irreversible blind spots.
- Scan-based Search (e.g., RAP, ZoomEye) ensures exhaustive coverage via rigid grid scanning but is semantically agnostic. This leads to computational redundancy on background regions and fractures object semantics due to rigid partitioning.

The core challenge is bridging the gap between the efficiency of expert guidance and the robustness of exhaustive search without compromising semantic integrity.

Methodology: CVSearch

CVSearch is a training-free, adaptive framework that empowers MLLMs with cognitive, human-like visual search capabilities. It operates on an Assess-then-Search workflow, dynamically alternating between non-selective global perception and selective serial attention based on task difficulty. The framework is underpinned by three key innovations:

1. Cognitive-Driven Adaptive Switching Mechanism

Inspired by human cognitive control, CVSearch first performs a Global Assessment to quantify information sufficiency ( $c_q$ ).

If global information is sufficient, the model provides a direct answer.
If insufficient, it triggers Visual Expert-Assisted Search (using SAM 3) for rapid localization.
Crucially, if the expert fails (i.e., fails to generate valid proposals), the system does not terminate. Instead, it interprets this failure as a signal to activate Scene-aware Scanning, ensuring a seamless transition from rapid localization to comprehensive exploration.

2. Semantic Guided Adaptive Patching (SGAP)

To address the semantic fragmentation inherent in rigid grid scanning, CVSearch introduces SGAP within the Scene-aware Scanning phase.

Mechanism: Instead of fixed grids, SGAP utilizes visual features extracted from the expert (even when localization fails) to perform adaptive clustering. It over-segments the image into atomic superpixels (via SLIC) and then clusters them into semantically coherent regions using Agglomerative Clustering constrained by a region adjacency graph.
Optimization: The number of clusters ( $k$ ) is dynamically optimized to minimize a cost function that penalizes spatial overlap while maximizing clustering quality (silhouette score).
Benefit: This ensures image patches align with object contours rather than cutting through them, preserving semantic integrity.

3. Dynamic Bottom-Up Search Strategy

To overcome error propagation inherent in top-down search methods, CVSearch employs a Dynamic Bottom-Up Search strategy driven by a Visual Complexity Prior.

Visual Complexity ( $c_v$ ): Quantifies the exploration value of a patch based on the dispersion of its atomic features. High complexity indicates semantically rich foregrounds; low complexity indicates redundant backgrounds.
Search Flow: The search initiates at the deepest layer (leaf nodes) where detailed local information is available, aggregating evidence upwards.
Pruning & Prioritization: Nodes with low visual complexity are pruned. Within each layer, nodes are sorted by a priority value ( $c_x$ ) that weights visual complexity, existence confidence, and aggregated child priorities.
Iterative Refinement: If the target remains unconfirmed after a full search, the top-ranked candidate from the first layer is fed back into the pipeline for iterative refinement, allowing the model to recover from initial failures.

Key Contributions

Cognitive Hierarchical Framework: CVSearch is the first training-free framework to unify the efficiency of visual expert-assisted search with the robustness of semantic-aware scanning via a failure-aware switching mechanism.
Semantic-Preserving Granularity: The proposed SGAP repurposes visual expert features to construct semantically consistent image patches, effectively overcoming the fragmentation caused by rigid grids.
Robust Bottom-Up Exploration: The dynamic bottom-up search strategy prevents error propagation typical of top-down methods, significantly improving the perception of small objects.
State-of-the-Art Performance: Extensive experiments demonstrate that CVSearch achieves SOTA accuracy on HR benchmarks while substantially improving search efficiency compared to scan-based baselines.

Experimental Results

The authors evaluated CVSearch on a comprehensive suite of high-resolution benchmarks, including V Bench*, HR-Bench (4K/8K), MME-RealWorld-Lite, TreeBench, and FineRS-4K.

Performance Gains: Integrating CVSearch with open-source MLLMs (e.g., LLaVA-OV-7B, Qwen2.5-VL-7B, InternVL2.5-8B) yielded substantial improvements. For instance, applying CVSearch to InternVL2.5-8B boosted overall accuracy on HR-Bench 8K by +20.2% and on V* Bench by +16.2% (reaching 91.6).
Comparison with SOTA: CVSearch outperformed both expert-assisted methods (e.g., DyFo) and scan-based methods (e.g., ZoomEye, RAP) across all benchmarks. On V* Bench, it surpassed the strong expert-assisted DyFo (81.2) with a score of 91.6.
Efficiency: While introducing some overhead compared to a vanilla single-pass model, CVSearch significantly outperformed existing scan-based frameworks in both accuracy and throughput. On HR-Bench 4K, it achieved a throughput of 3.77 samples/minute, nearly 3x faster than ZoomEye and RAP, while maintaining higher accuracy.
Scalability: The framework demonstrated consistent performance gains across models of varying sizes (2B to 32B) and generations (Qwen2.5-VL to Qwen3-VL), proving its effectiveness even for highly advanced backbones.

Significance and Claims

The paper claims that CVSearch resolves the critical efficiency-robustness dilemma in high-resolution visual perception by reformulating visual search as a cognitive, hierarchical decision-making process. By mimicking human visual search mechanisms—specifically the interplay between non-selective global gist extraction and selective serial attention—the framework allows MLLMs to capture fine-grained details with minimal computational redundancy.

The authors emphasize that this work steers the field toward more efficient, scalable, and accessible high-resolution multimodal understanding. By prioritizing adaptive resource allocation and mitigating computational redundancy through semantic-aware pruning and bottom-up exploration, CVSearch enables existing MLLMs to perform complex visual reasoning tasks without the need for extensive retraining or reinforcement learning pipelines.

Limitations Acknowledged:
The paper modestly notes two primary limitations:

Computational Latency: While faster than scan-based methods, the iterative search process is inherently slower than vanilla single-pass inference or lightweight expert-assisted baselines.
Visual Expert Dependency: The current implementation relies exclusively on SAM 3 for region proposals; the generalization capability with different types of visual experts remains unexplored.