Technical Summary: VideoSearcher

Problem Definition

Video understanding has traditionally operated under a closed-evidence paradigm, where models assume all necessary information resides within the input video frames. While effective for spatio-temporal perception, this approach is insufficient for real-world scenarios where videos often provide only visual anchors (e.g., a logo, a person, or a landmark) while the final answer requires retrieving external knowledge from open-web sources.

This gap defines the emerging task of Video Deep Research (VDR). VDR requires agents to:

1. Localize relevant moments and fine-grained visual details across evolving temporal dynamics.
2. Ground these visual cues to construct multimodal queries.
3. Retrieve evidence from open-web sources (text, images, news).
4. Synthesize answers by integrating video-grounded evidence with external knowledge.

Existing multimodal search agents are primarily designed for static images or rely on text-centric retrieval that discards crucial visual information during the search process. Furthermore, current VDR benchmarks lack rigorous evaluation of open-world video information grounding and multimodal search-based reasoning.

Methodology: VideoSearcher

To address these limitations, the authors propose VideoSearcher, a closed-loop agentic framework that empowers Vision-Language Models (VLMs) with multi-tool reasoning for VDR.

1. Agentic Framework and Toolset

VideoSearcher unifies temporal localization, spatial focusing, and multimodal search within a single reasoning trajectory. The agent operates with five specific tools:

• choose_frames: Selects a coarse temporal interval.
• find_frame: Localizes a specific key frame.
• zoom_in: Inspects a specific region (bounding box) within a frame.
• web_search: Retrieves textual web evidence.
• image_search: Performs visual retrieval using a frame or cropped region.

The agent iteratively narrows the video timeline, focuses on informative regions, retrieves multimodal evidence, and synthesizes the final answer.

2. Data Construction Pipeline

The authors constructed a video-centric data synthesis pipeline to generate high-quality training data:

• Entity-driven Acquisition: Core visual entities are extracted from knowledge-intensive image-text QA datasets (e.g., FVQA, DeepEyes) and cross-validated using large models (Qwen3.5, Kimi-K2.5) to ensure reliability. These entities serve as targets to retrieve relevant videos.
• Normalization & Alignment: Videos are resampled to 1 FPS with frame markers. Original QA pairs are rewritten to align with the video context.
• Hard Sample Mining: Samples that baseline models consistently fail to solve are retained to emphasize temporal localization and external evidence acquisition.
• Trajectory Synthesis: Multi-tool reasoning trajectories are synthesized using a hierarchical approach, followed by rigorous filtering to remove hallucinations, broken reasoning chains, and invalid tool invocations.

3. Bi-branch Sequence Policy Optimization (BiSPO)

To optimize long-horizon reasoning trajectories, the authors introduce BiSPO, a reinforcement learning (RL) algorithm built upon GSPO.

• Decoupled Optimization: BiSPO separates the optimization of answer accuracy from tool invocation behavior. This prevents the model from learning to guess answers correctly while using tools inefficiently, or vice versa.
• Dual Reward Branches:
  • Accuracy Branch: Evaluates answer correctness, format compliance, and tool-dependency constraints.
  • Tool Behavior Branch: Only active for trajectories with correct answers. It employs a bell-shaped gated reward that encourages sufficient video observation and external retrieval while penalizing excessive or redundant tool calls.
• Sequence-Level Clipping: To handle the instability of token-level clipping in long trajectories, BiSPO applies sequence-level importance sampling and clipping.

4. Benchmark: VideoSearch-QA (VSQA)

The authors introduce VideoSearch-QA, the first benchmark designed specifically to evaluate open-world video information grounding and multimodal search-based reasoning. It covers diverse domains (landmarks, geography, culture, sports) and requires agents to capture visual information in dynamic videos to perform open-web search and evidence-based reasoning.

Key Contributions

1. VideoSearcher Framework: The first closed-loop agentic agent integrating temporal localization, spatial focusing, and multimodal search for dynamic videos.
2. BiSPO Algorithm: A dual-branch RL algorithm that decouples answer accuracy from tool invocation, providing stable learning signals for knowledge-intensive reasoning.
3. VideoSearch-QA Benchmark: A manually annotated benchmark (278 samples) for evaluating VDR agents in open-world scenarios.
4. Data Pipeline: A rigorous pipeline for synthesizing high-quality, multi-tool reasoning trajectories from existing image-text QA datasets.

Experimental Results

Extensive experiments were conducted on VideoDR, VideoSearch-QA, and various multimodal search and video understanding benchmarks.

• Search-Oriented Benchmarks: VideoSearcher significantly outperforms prior open-source agentic baselines (including Qwen3-VL, Webwatcher, MMSearch-R1) and approaches the performance of proprietary models (GPT-4o, Gemini-3-Pro) on VideoDR and VSQA. Notably, it generalizes well to general multimodal search benchmarks (MMSearch, HR-MMSearch) despite being trained primarily on video-based data.
• General Video Understanding: On benchmarks like MMVU, TempCompass, and VideoMMMU, VideoSearcher maintains competitive reasoning abilities, demonstrating that the agentic training does not degrade general video understanding capabilities.
• Ablation Studies:
  • RL Design: BiSPO with the full bell-shaped tool reward outperforms single-branch variants (GRPO, GSPO) and monotonic tool rewards, confirming the necessity of decoupled optimization and the bell-shaped reward design.
  • Tool Necessity: Removing either localization tools or search tools leads to significant performance drops, proving that visual grounding and external retrieval are complementary and essential for VDR.

Significance and Claims

The paper claims that VideoSearcher represents a paradigm shift from closed-context perception to open-world evidence exploration in video understanding. By formalizing VDR and providing the first dedicated benchmark and agentic framework, the work addresses the critical bottleneck of deploying VLMs in real-world applications where videos serve as entry points for retrieving and verifying information in expansive web environments.

The authors position VideoSearcher as a step toward robust video understanding that requires both global context modeling and fine-grained spatio-temporal perception, enabling agents to progressively ground visual clues, retrieve relevant evidence, and synthesize answers in complex, open-world scenarios.

Limitations

The authors acknowledge several limitations:

• Dependency on External Tools: Answer quality relies on the latency, coverage, and stability of image/web search APIs.
• Computational Cost: Online RL with multi-turn tool use is computationally expensive due to long contexts and repeated rollouts.
• Scope: The current focus is on search-oriented visual/video QA. It may not fully cover tasks requiring dense temporal localization, audio understanding, embodied interaction, or specialized domain knowledge, particularly given the 1-fps frame representation used in training.
• Evaluation: Open-ended evaluation relies on LLM judges, which may introduce noise for ambiguous answers.