Technical Summary: Perception First: A Frontier Native-Video Model with Self-Consistency for Implicit Video Question Answering

1. Problem Definition

This work addresses the ImplicitQA / VRR-QA benchmark, a multiple-choice video question answering (VQA) task designed to test "implicit reasoning." Unlike standard VQA where answers are often visible in a single frame, ImplicitQA questions require inference across discontinuous frames based on spatial layout, motion, depth, viewpoint, causality, and social context. The benchmark consists of a validation split (1,001 questions) and a hidden test split (172 questions). The core challenge is that the answer cannot be read from any single frame; it must be synthesized from the video's temporal and spatial dynamics.

2. Methodology

The authors adopt a training-free approach, treating the task as an inference and test-time compute problem. The methodology relies on four pillars: base-model selection, video presentation, prompting strategies, and test-time aggregation.

2.1. Pipeline Architecture

The system employs a modular pipeline separating data loading, model inference, and result aggregation:

• Data Layer: Loads trimmed video clips (mean duration 16.6s), uniformly samples frames for open models, and optionally attaches Whisper-generated audio transcripts.
• Model Layer:
  • Open-Source: Uses vLLM to serve models like Qwen2.5-VL, Qwen3-VL, InternVL3, and Gemma-3. These models process multi-image inputs (sampled frames).
  • Frontier: Uses the Gemini 3.1 Pro API, which ingests native video (and audio) directly rather than sampled frames.
• Aggregation: The primary inference strategy is Self-Consistency, where $k=5$ independent reasoning paths are sampled at temperature $T=0.7$, and the final answer is determined by majority vote.

2.2. Experimental Strategy

The authors conducted a systematic ablation study on the validation split to select design decisions before committing to the hidden test set. Key variables tested included:

• Base Models: Scaling from 7B to 32B parameters (Qwen series) and utilizing frontier models.
• Prompting: Chain-of-thought (CoT), question decomposition, "describe-then-reason" cascades, spatial layout prompts, and depth-cue injection.
• Input Modalities: Frame sampling vs. native video ingestion; resolution scaling; audio integration.
• Ensembling: Multi-model ensembling and category routing.

3. Key Findings and Contributions

3.1. The "Perception-First" Hypothesis

The central finding of the paper is that the ImplicitQA benchmark is perception-bound rather than reasoning-bound.

• Reasoning Augmentations are Harmful: Strategies typically used to boost reasoning (e.g., complex decomposition, causal scaffolding, "describe-then-reason" cascades) were found to be neutral-to-harmful. Specifically, forcing a "describe-the-layout-first" or depth-cue prompt on the strongest model decreased accuracy by 5.8 points.
• The Bottleneck is Grounding: Error analysis revealed that the model's logical deduction (chain-of-thought) was often fluent and valid, but built on mis-perceived premises (e.g., incorrect depth ordering, egocentric vs. screen reference frame confusion, or counting errors due to undersampling). The model needs a better percept, not a better procedure.

3.2. The Efficacy of Native Video and Self-Consistency

• Native Video Ingestion: The most significant performance lever was the use of a frontier model (Gemini 3.1 Pro) that ingests native video. This provided the necessary perceptual capability that frame-sampling models lacked.
• Self-Consistency as Denoising: Self-consistency ($k=5$) acted as a "denoising" mechanism for strong base models, improving accuracy by +2.6 points on the 32B model and contributing to the final frontier score. Conversely, it amplified errors on weaker models (e.g., the 7B model).
• Open-Source Ceiling: The best open-source configuration (Qwen3-VL-32B-AWQ with self-consistency) achieved 58.5% accuracy on the test set.

3.3. Per-Category Error Analysis

A granular analysis of the frontier system's performance across nine reasoning categories revealed a stark divide:

• Hardest Categories (Perceptual): Relative depth & proximity (60%), Viewpoint & visibility (60%), and Inferred counting (60%). These are low-level perception tasks. Relative depth is the most frequent category (26.6% of the dataset) and the dominant factor in overall accuracy.
• Easiest Categories (Reasoning): Social interaction (100%) and Causal/motivational reasoning (95%). These high-level reasoning tasks are nearly solved by current models.
• Conclusion: The difficulty lies in low-level spatial perception, not complex reasoning.

4. Results

The final system, utilizing Gemini 3.1 Pro with 5-way self-consistency and native video input, achieved:

• Test Average Accuracy: 81.18%
• Test Macro-Average Accuracy: 78.85%

This performance surpasses the prior best result (80.85%) and approaches the non-expert human baseline (83.0% / 85.6%). The system was fully cached for exact reproduction, with a total inference cost of approximately $11.8.

5. Significance and Claims

The paper claims that the gap between human and machine performance on ImplicitQA is a perception/world-model gap, not a reasoning-search gap.

• Methodological Insight: Methods that benefit reasoning-bound tasks (tree search, decomposition, state tracking) are ineffective or detrimental when the bottleneck is perception.
• Practical Implication: For perception-bound benchmarks, the most effective strategy is selecting a model with superior native perceptual capabilities (specifically native video understanding) and applying lightweight test-time denoising (self-consistency).
• Limitations: The authors note that the 172-question test split introduces noise, and further attempts to improve performance via higher resolution, higher frame rates, or specialized depth prompts yielded null or negative results. They conclude that the frontier model is likely near its perceptual ceiling for this specific benchmark, and no legitimate path to surpass the non-expert human baseline was found within their budget and constraints.