Technical Summary: ACT2SEE – Emergent Active Visual Perception for Video Reasoning

Problem Statement

Current Vision-Language Models (VLMs) performing video reasoning typically rely on static initial frames or fixed frame-per-second (fps) sampling. This static input paradigm restricts the model's ability to incorporate essential dynamic information as the reasoning process evolves. While recent methods attempt to augment Chain-of-Thought (CoT) reasoning with additional frame information (e.g., predetermined keyframes or visual tool calls), they face two primary limitations:

1. Suboptimal CoT Quality: Existing methods often derive frame-augmented CoTs from automated VLM-generated ground truths without rigorous human verification, leading to noisy or inconsistent reasoning traces.
2. Inability to Synthesize Hypothetical Scenarios: Current approaches cannot generate new visual frames within the CoT to address counterfactual or hypothetical questions (e.g., "What would happen if the temporal order were switched?"). They are limited to retrieving existing visual evidence, preventing them from "imagining" scenarios not present in the original video.

Methodology

The authors introduce Act-to-See (ACT2SEE), a framework designed to endow VLMs with active visual perception. This capability allows models to dynamically decide when and how to gather new visual information—either by retrieving existing frames or generating hypothetical ones—during the reasoning process.

1. Dataset Construction via Supervised Fine-Tuning (SFT)

The core of ACT2SEE is a high-quality SFT dataset featuring interleaved video-text CoTs. The construction pipeline involves:

• Frontier Model Prompting: A frontier VLM (Gemini 2.5 Pro) is prompted to perform video reasoning. Crucially, the model is instructed to output specialized tool call tokens (<retrieve> or <generate>) with specific queries whenever it determines existing visual information is insufficient.
• Tool Execution:
  • Retrieval: Uses TFVTG (Training-Free Video Temporal Grounding) to search the full video at a higher sampling rate (3 fps) to find relevant frames matching the query.
  • Generation: Uses Stable Diffusion 3.5 Large to synthesize new frames based on the query, conditioned on a retrieved frame to ensure visual consistency.
• Two-Round Completion: The process is split into two rounds. In Round 1, the model generates reasoning up to the tool call. The tool executes, and the new frame is inserted. In Round 2, the model completes the reasoning trace using the new visual evidence.
• Quality Control: To ensure high-quality training data, the generated CoTs are filtered against human-annotated ground-truth CoTs from benchmarks like MINERVA, CausalVQA, and Social Genome. Traces are retained only if their text embedding similarity (BGE M3) to the human ground truth exceeds 80%. CoTs leading to incorrect answers are regenerated.

2. Training and Inference

• Training: A base model (Qwen3-VL-8B-Thinking) is fine-tuned using the constructed interleaved dataset. The loss function computes token-level losses over the entire rollout, including the text reasoning and the tool call queries, but excluding the actual retrieved/generated image tokens.
• Inference: The trained model follows the same algorithmic flow. It generates a response; if a <retrieve> or <generate> token is detected, the corresponding offline tool is executed, the new frame is injected into the context, and the model continues reasoning to produce the final answer.

Key Contributions

1. Emergent Active Perception: ACT2SEE enables VLMs to actively interleave video frames within text CoTs at inference time, allowing them to dynamically search for missing factual evidence or synthesize counterfactual scenarios.
2. High-Quality Interleaved Dataset: The authors curate a dataset of 3,373 reasoning traces where 47.67% contain retrieved or generated frames. This dataset is rigorously filtered against human annotations to ensure reasoning quality, addressing the noise issues in previous automated CoT generation.
3. Dual Capability (Retrieval + Generation): Unlike prior works that focus solely on retrieval or static keyframes, ACT2SEE integrates both retrieval of existing video content and generative synthesis of hypothetical visual evidence within the reasoning loop.

Experimental Results

ACT2SEE was evaluated on five diverse video reasoning benchmarks: Video-MME, VideoEspresso, EgoNormia, VCR-Bench, and ViTIB.

• Performance Gains: ACT2SEE establishes new state-of-the-art results on VideoEspresso and ViTIB. It outperforms comparable or larger open-source models (e.g., InternVL2.5-8B, Video-LLaMA3-7B, Qwen3-VL-8B-Thinking) and even surpasses the closed-source GPT-4o on VideoEspresso, despite using fewer frames.
• Comparison with SOTA CoT Methods: The model outperforms other video-text interleaved CoT methods like Chain-of-Shot, FrameMind, and ReWatch-R1.
• Ablation Studies:
  • Data Source: Using human-annotated ground truths for filtering (rather than VLM-generated CoTs) significantly improves performance.
  • Tool Usage: Combining both retrieval and generation yields better results than using either tool in isolation.
  • Ground Truth in Prompting: Excluding ground-truth CoTs from the initial prompting phase increases the rate of tool calls, leading to better downstream performance.
• Efficiency: ACT2SEE achieves higher accuracy with lower latency and fewer FLOPs compared to the baseline when using half the number of input frames, demonstrating Pareto dominance.

Significance and Claims

The paper claims that ACT2SEE represents a significant advancement in enabling VLMs with active visual perception for complex video reasoning. By empowering models to actively determine when to search for or synthesize visual evidence, the framework overcomes the limitations of static inputs and suboptimal reasoning traces.

The authors highlight an emergent capability: at inference time, the model spontaneously decides to retrieve or generate frames to resolve ambiguities or answer counterfactual questions, a behavior not explicitly programmed but learned through the SFT process. This approach moves video reasoning toward more causally grounded and dynamic understanding, bridging the gap between static perception and the nuanced spatiotemporal dynamics required for complex real-world scenarios. The work suggests that high-quality, human-verified interleaved data is critical for unlocking these capabilities, rather than relying solely on automated data synthesis.