Technical Summary: LongVT

Problem Statement

Large Multimodal Models (LMMs) have demonstrated potential for video reasoning using textual Chain-of-Thought (CoT). However, they remain vulnerable to hallucinations, particularly when processing long-form videos (>15 minutes) where evidence is sparse and temporally dispersed. Existing approaches, often relying on R1-style paradigms (Supervised Fine-Tuning with textual CoT followed by Group Relative Policy Optimization), are largely language-centric. They fail to adaptively capture key visual evidence, often missing fine-grained or decisive moments critical for reasoning. Furthermore, current benchmarks frequently suffer from data contamination and option bias, failing to rigorously test "Video Segment-in-a-Haystack" capabilities where the answer lies within a narrow, specific window of hours-long footage.

Methodology

The paper introduces LongVT, an end-to-end agentic framework designed to enable "Thinking with Long Videos" via Interleaved Multimodal Chain-of-Tool-Thought (iMCoTT).

Core Mechanism: iMCoTT

Inspired by human viewing habits (global skimming followed by detailed inspection), LongVT allows LMMs to natively interleave reasoning with on-demand temporal retrieval. Instead of blindly rephrasing text, the model:

1. Global Skim: Performs a coarse preview of the video to formulate a hypothesis about relevant temporal windows.
2. Tool Invocation: Calls a native crop_video(start_time, end_time) tool to resample finer-grained frames from the hypothesized segment.
3. Verification & Self-Correction: Analyzes the retrieved clip. If the evidence is insufficient or the localization is incorrect, the model engages in self-reflection, refines the time window, and calls the tool again. This loop continues until the answer is grounded in retrieved visual evidence.

Data Suite: VideoSIAH

To address the scarcity of fine-grained QA data for long-video reasoning, the authors curated VideoSIAH, a large-scale dataset comprising:

• Training Data: 247.9K samples for cold-start Supervised Fine-Tuning (SFT) (including tool-augmented traces), 1.6K samples for Agentic Reinforcement Learning (RL), and 15.4K samples for Agentic Reinforcement Fine-Tuning (RFT).
• Evaluation Benchmark (VideoSIAH-Eval): 652 carefully curated open-ended QA pairs with human-in-the-loop validation, specifically designed to test evidence-sparse reasoning in videos averaging ~1,688 seconds.

Three-Stage Training Strategy

1. Cold-Start SFT: Teaches the base LMM (Qwen2.5-VL-7B) to propose temporal windows, invoke the crop tool, and compose multimodal evidence. This stage is critical to prevent RL collapse due to the model's initial inability to localize windows or integrate tool outputs.
2. Agentic RL: Optimizes the model using a joint answer-temporal grounding reward. Unlike prior works that reward answer correctness or temporal alignment separately, this reward function couples the two, explicitly encouraging exploratory rollouts that improve temporal localization while preserving answer correctness.
3. Agentic RFT: Distills high-quality trajectories from the RL stage back into supervised data. This stabilizes the agentic behaviors learned during RL and consolidates fine-grained temporal localization and multi-step reasoning.

Key Contributions

1. End-to-End Agentic Paradigm: LongVT is the first framework to natively interleave multimodal tool-augmented CoT with on-demand clip inspection for hours-long videos, enabling LMMs to perform active, evidence-seeking reasoning rather than passive frame consumption.
2. VideoSIAH Dataset & Benchmark: The construction of a scalable data pipeline producing diverse, high-quality tool-integrated reasoning traces and a dedicated evaluation benchmark that mitigates data contamination and option bias found in existing benchmarks.
3. Training Strategy Validation: The paper systematically validates that a three-stage pipeline (SFT + RL + RFT) with a joint reward function yields consistent gains over SFT or RL alone, establishing a new state-of-the-art baseline for long-video reasoning.

Experimental Results

Evaluated on four benchmarks (VideoMME, VideoMMMU, LVBench, and VideoSIAH-Eval) under both sparse (64 frames) and dense (512/768 frames) sampling regimes:

• Performance: LongVT-7B-RFT achieves state-of-the-art performance among open-source video-centric LMMs. On the challenging VideoSIAH-Eval, it reaches a score of 42.0, outperforming the second-best open-source model by 6 points.
• Gap Narrowing: The best LongVT checkpoint narrows the gap with proprietary models (e.g., GPT-4o) to within roughly 4 points on average.
• Efficiency: Despite multi-turn tool interactions, LongVT does not incur additional inference latency compared to single-turn baselines; in some cases, it is faster by avoiding hallucination-driven verbose generation.
• Ablation Studies: Results confirm that self-curated iMCoTT data is crucial for SFT, the joint reward function outperforms separate rewards, and the full three-stage pipeline is necessary for optimal performance.

Significance and Claims

The paper claims that LongVT successfully transforms long-video understanding from a passive task into an active, evidence-seeking process. By grounding each reasoning step in retrieved visual evidence via native tool calling, the framework significantly mitigates hallucinations and enhances temporal localization. The authors posit that this "Thinking with Long Videos" paradigm, which mirrors human cognitive strategies, is a necessary evolution for reliable long-form video reasoning. The work establishes that open-source models can achieve robust, human-aligned reasoning capabilities through structured tool-augmented training and high-quality, fine-grained data, without relying on external retrievers or auxiliary expert models.