Video Active Perception: Effective Inference-Time Long-Form Video Understanding with Vision-Language Models

Technical Summary: Video Active Perception (VAP)

Problem Statement
Large Vision-Language Models (VLMs) have advanced multimodal tasks like video question answering (QA), yet they face significant hurdles when applied to long-form videos. The authors identify two primary challenges:

1. Frame Effectiveness: Standard inference often relies on uniform sampling (e.g., 1 frame per second). This approach is suboptimal because it treats all video regions equally, potentially selecting irrelevant frames while missing information-dense, task-relevant segments. Empirical results show that performance often plateaus or even degrades as the number of uniformly sampled frames increases.
2. Frame Efficiency: Processing long videos generates a prohibitive number of tokens (nearly four million for a single hour), hindering the practical deployment of VLMs in real-world scenarios like autonomous driving or patient monitoring.

Existing solutions often rely on iterative agent-based reasoning, video captioning, or complex data structures (e.g., graphs or trees), which introduce high latency, depend on the quality of intermediate captioning models, or require extensive computational resources.

Methodology: Video Active Perception (VAP)
The paper introduces Video Active Perception (VAP), a training-free, inference-time method designed to enhance both the effectiveness and efficiency of frame selection for long-form video QA. Inspired by active perception theory—which posits that intelligent agents acquire data that differs from their expectations to maximize information gain—VAP reframes keyframe selection as a data acquisition process guided by prior world knowledge.

The method operates in three main stages without fine-tuning the VLM:

1. A Priori World Knowledge Generation:

   • VAP utilizes a lightweight, pre-trained text-conditional video generation model (specifically CogVideoX) to represent prior visual knowledge.
   • The system takes a small subset of uniformly sampled initial frames (e.g., 32 frames), along with the user's question and possible answers, as conditional signals.
   • Using a 3D Causal VAE and a frame interpolation module (RIFE), the model generates latent representations of the entire video dynamics, predicting how the video is expected to unfold based on the initial context and the query.

2. Data Acquisition (Frame Selection):

   • The system encodes all actual video frames into the same latent space using the same 3D VAE encoder.
   • It then compares the latent representations of the real frames against the generated (expected) frames.
   • Selection is based on divergence: frames where the real video dynamics differ most significantly from the generation model's predictions are considered "surprising" and thus most informative.
   • The method selects the $n$ frames (e.g., 32) with the lowest cosine similarity to their generated counterparts.

3. VLM Inference:

   • The selected keyframes are fed into a flagship VLM (e.g., GPT-4o, Gemini 1.5 Pro, LLaVA-OV) along with the question and answers to perform the final inference.
   • This process is agnostic to the specific VLM used and operates in a single selection round, avoiding iterative loops.

Key Contributions

• Training-Free Framework: VAP requires no fine-tuning of large VLMs, making it immediately applicable to existing models.
• Novel Selection Mechanism: Unlike previous methods that rely on captions or iterative agents, VAP leverages a generative model to create a "prior" and selects frames based on the discrepancy between expectation and reality.
• Unified Approach: It simplifies the selection process to a single round of comparison, eliminating the need for complex data structures or multiple LLM calls.
• Efficiency: It drastically reduces the token count required for inference by selecting only the most relevant frames.

Experimental Results
The authors evaluated VAP on five benchmark datasets: EgoSchema, NExT-QA, ActivityNet-QA, IntentQA, and CLEVRER.

• Performance: VAP achieved state-of-the-art zero-shot results across all datasets. For instance, it reached 68.1% on EgoSchema, 81.4% on NExT-QA, and 40.5% on CLEVRER.
• Reasoning Capabilities: VAP demonstrated superior performance on tasks requiring temporal and causal reasoning (e.g., explanatory and counterfactual questions in CLEVRER), outperforming baselines like VideoTree and VideoAgent.
• Efficiency Gains:
  • VAP achieved these results using significantly fewer frames than standard baselines. For example, on EgoSchema, it used 32 frames compared to the standard 180 frames (1 fps), resulting in a 5.6× improvement in frame efficiency.
  • In latency comparisons (iso-compute and iso-accuracy), VAP achieved better or comparable accuracy with lower latency than standard GPT-4o mini sampling.
• Robustness: Experiments showed that the method is robust to the specific set of initial frames used for generation and that increasing the number of selected frames beyond 32 yields diminishing returns.

Significance and Claims
The paper claims that VAP demonstrates the empirical effectiveness of leveraging active perception principles to improve long-form video understanding. By using a generative model to encode prior world knowledge and selecting frames that diverge from these expectations, the method successfully identifies pivotal moments (such as collisions or specific actions) that are crucial for answering complex questions.

The authors highlight that this approach offers a more intelligent data acquisition strategy for VLM inference, balancing high performance with computational efficiency. They conclude that integrating prior knowledge from diffusion models is a viable path toward enhancing the frame effectiveness and efficiency of long-form video QA, particularly for reasoning-heavy tasks. The work suggests that future advancements in video generation models could further empower VLMs with stronger spatio-temporal reasoning capabilities over very long videos.