💻 computer science

Video-o3: Native Interleaved Clue Seeking for Long Video Multi-Hop Reasoning

Video-o3 is a novel framework for long-video multi-hop reasoning that overcomes the limitations of uniform sampling by enabling native, iterative clue seeking through task-decoupled attention masking and a verifiable trajectory-guided reward, achieving state-of-the-art performance on benchmarks like MLVU and Video-Holmes.

Original authors: Xiangyu Zeng, Zhiqiu Zhang, Yuhan Zhu, Xinhao Li, Zikang Wang, Changlian Ma, Qingyu Zhang, Zizheng Huang, Kun Ouyang, Tianxiang Jiang, Ziang Yan, Yi Wang, Hongjie Zhang, Yali Wang, Limin Wang

Published 2026-05-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xiangyu Zeng, Zhiqiu Zhang, Yuhan Zhu, Xinhao Li, Zikang Wang, Changlian Ma, Qingyu Zhang, Zizheng Huang, Kun Ouyang, Tianxiang Jiang, Ziang Yan, Yi Wang, Hongjie Zhang, Yali Wang, Limin Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a mystery in a 30-minute movie, but you only have a few seconds to watch it. Most current AI models try to solve this by taking a quick, blurry snapshot of the whole movie every few seconds and then guessing the answer immediately. It's like trying to find a specific needle in a haystack by glancing at the whole pile once and hoping you saw it. You often miss the needle because it's hidden in the "hay" (the boring parts), or you get overwhelmed by too much information.

Video-o3 is a new AI framework that changes the game. Instead of glancing, it acts like a detective with a magnifying glass. Here is how it works, broken down into simple concepts:

1. The Detective's Approach (Native Interleaved Clue Seeking)

Instead of watching the whole movie and then guessing, Video-o3 watches a little bit, thinks, and then decides: "I need to zoom in on this specific 5-second part to see what's happening."

How it works: It asks the system to "crop" a specific segment of the video (like cutting a small piece of film out of the reel) to look at it closely.
The Loop: It repeats this process. Watch a bit $\rightarrow$ Think $\rightarrow$ Zoom in on a clue $\rightarrow$ Think again $\rightarrow$ Zoom in on another clue.
The Result: It builds the answer piece by piece, only looking at the parts of the video that actually matter, ignoring the rest.

2. The Problem of "Split Attention" (Task-Decoupled Attention Masking)

Imagine a detective who is trying to write a report while simultaneously looking at a crime scene. If they try to do both at the exact same time, they might get confused. They might look at the crime scene but write down something from their memory instead of what they are actually seeing. This is called "fake thinking."

The Fix: Video-o3 uses a special "masking" technique.
- When the AI is looking for clues (scanning the video), it is forced to ignore the final answer it might be writing. It focuses entirely on finding the evidence.
- When the AI is writing the answer, it is forced to ignore the raw video footage and focus only on the clues it just found.
The Analogy: It's like a student who is allowed to look at their textbook only during the "study" phase, and only allowed to look at their notes during the "test" phase. This prevents them from cheating or getting confused, ensuring their final answer is based on solid evidence.

3. The "Stop Sign" (Verifiable Trajectory-Guided Reward)

A detective could theoretically keep zooming in forever, looking at every single frame, which would take too long and cost too much money (computing power).

The Fix: Video-o3 is trained with a special reward system.
- If the AI finds the answer quickly and accurately, it gets a big "gold star" (reward).
- If the AI keeps zooming in unnecessarily after it already has the answer, it gets a "penalty" (the reward gets smaller).
The Analogy: It's like a game of "Hot and Cold." The AI learns to stop searching the moment it feels "hot" (has enough evidence). It learns to be efficient, stopping exactly when it has solved the puzzle, rather than wasting time looking at empty rooms.

4. The Training Ground (Seeker-173K)

To teach an AI to be a detective, you can't just give it a textbook; you need to give it thousands of practice cases where it has to hunt for clues.

The Dataset: The researchers created a massive dataset called Seeker-173K. This contains 173,000 examples of "video mysteries" where the AI is forced to practice the cycle of: Look $\rightarrow$ Zoom $\rightarrow$ Think $\rightarrow$ Look Again $\rightarrow$ Solve.
The Result: Because it practiced so much on these specific "clue-hunting" tasks, it became much better at solving complex video questions than previous models.

The Bottom Line

Video-o3 is a smarter way for computers to watch long videos. Instead of trying to memorize the whole movie at once, it acts like a human detective: it scans the scene, zooms in on the important details, connects the dots, and stops as soon as it has the proof. This makes it much faster, more accurate, and better at solving complex puzzles hidden in long videos.

Performance: In tests, this approach allowed the AI to solve video puzzles with significantly higher accuracy (e.g., 72.1% on one major benchmark) compared to older methods that just "glanced" at the video.

Technical Summary: Video-o3

Problem Statement

Existing Multimodal Large Language Models (MLLMs) for long-video understanding predominantly rely on uniform frame sampling followed by single-turn inference. This approach struggles to identify sparse yet critical evidence within extensive redundant background content, leading to significant computational overhead and degraded reasoning accuracy. While early "clue seeking + answer reasoning" prototypes exist, they often depend on hand-crafted heuristics, lack end-to-end optimization, and decouple clue seeking from reasoning, failing to scale to complex, long-form videos requiring joint multi-hop reasoning. Furthermore, native multi-turn tool invocation introduces two core challenges: attention dispersion caused by the heterogeneity of reasoning and tool-calling tasks within a shared context, and contextual inefficiency where uncontrolled tool interactions lead to excessive token consumption and redundant exploration.

Methodology

The authors introduce Video-o3, a framework designed for native interleaved clue seeking and multi-hop reasoning in long videos. The system enables the model to iteratively discover salient visual clues, perform fine-grained inspection of key segments via a VideoCrop tool, and adaptively terminate the search once sufficient evidence is acquired.

1. Task-Decoupled Attention Masking (TDAM)

To mitigate attention dispersion induced by heterogeneous tasks (clue discovery vs. answer reasoning), the authors propose TDAM. This mechanism imposes explicit visibility constraints during Supervised Fine-Tuning (SFT):

Clue Seeking Phase: The model is restricted to attending solely to the global video input, forcing it to learn planning strategies based on the global context.
Answer Reasoning Phase: The global overview is masked, compelling the model to derive answers exclusively from high-resolution tool observations.
This strategy is applied to only 10% of the training data to balance decoupled expertise with the need for holistic reasoning, effectively isolating the training of these two modes while maintaining a shared global context.

2. Verifiable Trajectory-Guided Reward (VTGR)

To control context length growth and encourage efficient exploration, a novel reward function is introduced for Reinforcement Learning (RL). The reward $R$ is a composite of answer correctness, format validity, and a trajectory-guided multiplier $\beta$ :
$R = r_a \cdot (1 + \beta) + r_f$

Hybrid Clue Score ( $S_{clue}$ ): Incentivizes precise localization by rewarding trajectories that align with ground-truth intervals (using IoU, IoP, IoG) or encouraging diversity in "free exploration" samples.
Turn Decay Factor ( $\gamma$ ): Penalizes trajectories that exceed a reference number of turns, promoting agile termination and preventing redundant loops.
The model is optimized using the DAPO algorithm, a variant of GRPO, to maximize relative advantages within sampled groups.

3. Data Synthesis and Seeker-173K

To overcome data scarcity for training native interleaved tool invocation, the authors developed a scalable automated data synthesis pipeline. This pipeline processes "Video-Question-Answer" triplets through four stages: Clue Localization, Validity Verification, Trajectory Generation, and Logical Consistency Check. This process yields Seeker-173K, a high-quality corpus of 173,000 tool-interaction trajectories explicitly designed to cultivate robust native interleaved tool invocation capabilities.

Key Contributions

Task-Decoupled Attention Masking: A mechanism that disentangles clue localization from answer reasoning to prevent attention interference, enabling end-to-end models to concentrate on specific exploration stages while preserving global context.
Verifiable Trajectory-Guided Reward: A reward formulation that balances exhaustive exploration with computational efficiency, encouraging the model to localize clues accurately and terminate reasoning precisely when evidence is sufficient.
Seeker-173K Dataset: A large-scale, high-quality training corpus comprising 173k tool-interaction trajectories, constructed via an automated synthesis pipeline to support supervised and reinforcement learning for long-video reasoning.

Experimental Results

Extensive evaluations across diverse long-video understanding and reasoning benchmarks demonstrate that Video-o3 substantially outperforms state-of-the-art methods:

MLVU: Achieved 72.1% accuracy (SFT+RL), surpassing previous bests.
Video-Holmes: Achieved 46.5% accuracy, demonstrating strong multi-hop evidence-seeking capabilities.
LVBench: Achieved 47.6% accuracy.
Temporal Grounding: On the Charades-STA benchmark, Video-o3 achieved an mIoU of 60.7, significantly outperforming baselines like LongVT (27.2) and Qwen2.5-VL (43.6).
Efficiency: Compared to decoupled iterative reasoning methods, Video-o3 reduced inference time by 32.9% (vs. LOVE-R1) and 46.0% (vs. VideoChat-R1.5) while using fewer visual tokens, leveraging a unified shared context to eliminate redundant computation via KV cache.

The framework also demonstrated generalization to smaller base models (Qwen3-VL-4B), yielding consistent improvements across all tested benchmarks.

Significance and Claims

The paper claims that Video-o3 establishes a new state-of-the-art for long-form video understanding by validating the effectiveness of native tool invocation in these scenarios. By enabling models to actively seek task-relevant evidence through iterative, multi-turn interactions rather than passively processing static inputs, Video-o3 addresses the fundamental limitations of uniform sampling and single-turn inference. The work underscores the transformative potential of native tool-use paradigms in advancing the next generation of agentic MLLMs, specifically highlighting the ability to decompose complex queries into manageable sub-goals and resolve ambiguities through robust multi-hop reasoning chains. The authors emphasize that their approach successfully balances the trade-off between exploration coverage and reasoning efficiency, a critical requirement for scalable long-video analysis.