🤖 AI

VideoThinker: Building Agentic VideoLLMs with LLM-Guided Tool Reasoning

VideoThinker addresses the challenge of long-form video understanding by training an agentic VideoLLM on a large-scale synthetic dataset, where a powerful language model generates multi-step tool reasoning trajectories in caption space that are subsequently grounded back to video frames, enabling adaptive temporal exploration and superior performance without requiring prior long-form video comprehension.

Original authors: Chenglin Li, Qianglong Chen, Feng Han, Yikun Wang, Xingxi Yin, Yan Gong, Ruilin Li, Yin Zhang, Jiaqi Wang

Published 2026-04-21

📖 4 min read☕ Coffee break read

CC0 1.0

Original authors: Chenglin Li, Qianglong Chen, Feng Han, Yikun Wang, Xingxi Yin, Yan Gong, Ruilin Li, Yin Zhang, Jiaqi Wang

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a mystery in a 30-hour movie marathon. You have a question: "What was the detective wearing when he found the clue in the kitchen?"

The Old Way (Standard VideoLLMs):
Imagine a tired assistant who tries to watch the whole movie at once. To save time, they only glance at 10 random snapshots from the entire 30 hours. They might miss the kitchen scene entirely, or they might mix up the detective's outfit from the beginning of the movie with the end. They guess based on those few blurry snapshots, often getting it wrong because they missed the crucial moment.

The "Agent" Way (Current LLM Agents):
Imagine a brilliant detective (a powerful text-based AI) who has never seen the movie. You hand them a script (text captions) and say, "Find the answer." The detective is smart and knows how to use tools. They say, "I need to check the kitchen scene!" But they can't see the video. They have to ask a second person (a separate video AI) to describe the kitchen scene to them. Then they ask again, "What about the hallway?" It's a slow, clunky conversation between two people where one is blind and the other is just a camera. They eventually get the answer, but the process is inefficient and the "blind" detective never really learns to see.

The New Way: VideoThinker
The paper introduces VideoThinker, a new kind of AI that combines the best of both worlds. Think of VideoThinker as a super-sleuth who learns to use a "Time Machine" and a "Magic Zoom Lens."

Here is how it works, using a simple analogy:

1. The Problem: The Needle in a Haystack

Long videos are like a giant haystack. Finding a specific moment (the needle) is hard. If you just scan the whole thing quickly, you miss details. If you look at every single second, it takes forever and your brain (the computer) gets tired.

2. The Solution: Learning to "Think with Videos"

VideoThinker doesn't just watch; it investigates. It is trained to use two special tools:

Temporal Retrieval (The Map): This tool scans the video's subtitles or audio to find where the interesting stuff might be. It's like asking, "Where in the movie does the detective go to the kitchen?" and getting a list of time stamps.
Temporal Zoom (The Magnifying Glass): Once it knows where to look, it zooms in on that specific 10-second clip. It doesn't just guess; it looks closely at the frames to see the details.

3. The Secret Sauce: The "Fake" Training

Here is the clever trick the researchers used. They knew that teaching an AI to use these tools is hard because you need a teacher who already knows how to solve long videos. But no one had that teacher yet!

So, they created a simulated training camp:

They took a video and turned it into a long text story (captions).
They asked a super-smart text AI (the "Teacher") to solve the mystery using the text story. The Teacher would say, "I need to check the kitchen," and "I need to zoom in on the clock."
They recorded the Teacher's steps.
The Magic Swap: They took those text steps and replaced the text descriptions with the actual video frames.
Now, the VideoThinker (the student) learned by watching the Teacher's "thought process," but with real video frames instead of just words.

It's like teaching a pilot to fly by having them watch a simulator where the instructor's voice is replaced by the actual view out the cockpit window. The student learns how to look and when to zoom, without needing a human to hold their hand.

4. The Result: Confidence and Speed

VideoThinker has a "confidence meter."

If the question is easy (e.g., "Is there a car in the video?"), it answers quickly by looking at a few frames.
If it's unsure (e.g., "What color was the car's license plate?"), it says, "I'm not 100% sure," and then activates its Time Machine and Zoom Lens to hunt down the exact moment and look closer.

Why This Matters

No More Guessing: It doesn't just guess based on random frames; it actively searches for the answer.
Efficiency: It doesn't waste time watching the whole movie. It jumps straight to the important parts.
Learning: It learns to "see" and "think" at the same time, rather than relying on a separate blind assistant.

In short: VideoThinker is like giving a detective a map and a magnifying glass, then training them to use those tools perfectly so they can solve mysteries in a 30-hour movie without getting lost or missing a single clue.

1. Problem Statement

Current Video Large Language Models (VideoLLMs) struggle with long-form video understanding. The primary limitations include:

Static Reasoning & Information Loss: Existing models typically rely on uniformly sampled frames, which fails to capture sparse, temporally distributed visual evidence. This leads to significant information loss and poor temporal localization.
Computational Cost: Processing hundreds of frames from long videos (e.g., >30 minutes) is computationally expensive.
The "Circular Dependency" in Agentic Training: Constructing high-quality agentic video data (where a model learns to use tools like retrieval and zoom) usually requires a model that already possesses strong long-video comprehension. This creates a chicken-and-egg problem: you need a smart agent to generate training data for a smart agent.
Limitations of Current Agentic Approaches:
- Proprietary models (e.g., Gemini-Pro) used for data synthesis lack emergent "thinking with videos" capabilities and require heavy prompt engineering.
- Existing open-source frameworks often treat the VideoLLM merely as a passive captioning tool, while the LLM (which cannot see) does the reasoning. This decouples visual perception from reasoning.

2. Methodology: VideoThinker

The authors propose VideoThinker, an agentic VideoLLM trained entirely on synthetic tool-interaction trajectories. The core innovation is converting the video reasoning problem into a caption space first, then grounding it back to video frames.

A. Core Architecture & Workflow

Caption-First Data Synthesis:
- Instead of feeding raw video frames to a powerful LLM (which lacks visual perception), the system first converts the entire video into rich textual captions using a standard VideoLLM.
- A powerful Agentic LLM (e.g., Qwen3) is then prompted to generate multi-step reasoning trajectories in this caption space. It decides which "tools" to call to answer a question based on the text captions.
Tool Design:
The system employs two complementary categories of tools:
- Temporal Retrieval Tools:
  - Clip Retrieval: Retrieves semantically relevant 10-second clips using embeddings (LanguageBind-Video).
  - Subtitle Retrieval: Uses Whisper to transcribe audio and retrieves relevant subtitle segments.
  - Subtitle Summary: Generates high-level summaries of the narrative flow.
- Temporal Zoom Tools:
  - Frame Zoom: Extracts raw frames from a specific interval for detailed visual inspection.
  - Subtitle Zoom: Extracts localized text for a specific interval.
  - Caption Zoom: (Used in synthesis) Generates text captions for a specific interval.
Synthetic Data Construction (The "Grounding" Step):
- The Agentic LLM generates a Chain-of-Thought (CoT) trajectory using Caption Zoom (text-based).
- Crucial Step: During dataset construction, the text outputs of Caption Zoom are replaced with actual video frames (via Frame Zoom).
- This creates a video-interleaved CoT dataset where the model learns to reason step-by-step, retrieving specific frames and subtitles, without the VideoLLM ever needing to "understand" the long video initially.
Training:
- The VideoLLM (student model, e.g., Qwen2.5-VL-7B) is fine-tuned on this synthetic dataset.
- The loss function minimizes the difference between the model's generated reasoning trajectory (including tool calls and frame observations) and the synthetic ground truth.
- This teaches the VideoLLM to actively retrieve and perceive key frames during reasoning, effectively internalizing the "thinking with videos" capability.

B. Adaptive Video Reasoning (Inference)

To balance efficiency and accuracy, VideoThinker uses a confidence-gated mechanism:

Direct Reasoning: For short videos (<600s) or when the model is confident, it samples a few frames and answers directly.
Tool-Augmented Reasoning: If the confidence score ( $\gamma$ ) is below a threshold ( $\tau$ ), the model triggers the agentic loop. It uses retrieval tools to find relevant intervals and zoom tools to inspect frames, refining its answer iteratively.

3. Key Contributions

Synthetic Agentic Dataset: The first method to construct a large-scale, high-quality agentic video-reasoning dataset without requiring a pre-existing long-video understanding model. It breaks the circular dependency by using LLMs in caption space and grounding them later.
VideoThinker Model: A single, end-to-end VideoLLM (7B parameters) that integrates dynamic reasoning, adaptive temporal exploration, and multi-step tool use. It elevates the VideoLLM from a passive tool to the core reasoning engine.
Tool Suite: A modular set of retrieval and zoom tools (Clip, Subtitle, Frame, Caption) that enable coarse-to-fine analysis of long videos.
Efficiency: The approach avoids the need for proprietary models (like Gemini-Pro) or complex multi-stage filtering, relying instead on open-source LLMs and synthetic data generation.

4. Experimental Results

The model was evaluated on four major long-form video benchmarks: MLVU, LVBench, VideoMME (Long), and LongVideoBench.

Performance Gains:
- MLVU: 54.8% (vs. 48.0% for Qwen2.5-VL-7B; +6.8% improvement).
- LVBench: 48.9% (vs. 38.3% for Qwen2.5-VL-7B; +10.6% improvement).
- VideoMME (Long): 53.7% (vs. 50.0% for Qwen2.5-VL-7B).
- LongVideoBench: 59.7% (vs. 56.7% for Qwen2.5-VL-7B).
Comparison:
- VideoThinker (7B) outperforms larger open-source models (e.g., Qwen2.5-VL-72B) and competitive closed-source models (e.g., GPT-4o, Gemini-1.5 Pro) on several metrics.
- It significantly outperforms "Language Agent" baselines where the VideoLLM is just a tool, proving that integrating reasoning into the VideoLLM itself is more effective.
Ablation Studies:
- Video Duration: The performance gap widens as video length increases. VideoThinker shows a +3.7% gain on VideoMME and +5.6% on LongVideoBench for videos >15 minutes compared to the baseline.
- Confidence Threshold: Adaptive reasoning (triggering tools only when confidence is low) yields the best trade-off between speed and accuracy.
- Retrieval Top-k: Retrieving multiple clips/subtitles (Top-3 or Top-5) generally improves performance over single-item retrieval.

5. Significance

Paradigm Shift: VideoThinker demonstrates that "Thinking with Videos" is achievable by training VideoLLMs on synthetic, tool-augmented trajectories. It moves beyond static frame analysis to dynamic, adaptive exploration.
Scalability: By using synthetic data generated via LLMs in caption space, the method bypasses the need for expensive human annotation or proprietary model distillation, making it scalable for diverse long-form video tasks.
Efficiency: The confidence-gated mechanism ensures that computationally expensive tool usage is only triggered when necessary, making the system practical for real-world deployment.
Generalization: The framework successfully bridges the gap between text-based reasoning and true video reasoning, enabling models to handle temporal sparsity and information loss in videos ranging from minutes to hours.