Technical Summary: Chain-of-Glimpse for Video Understanding

Problem Statement

Video understanding requires models to identify and reason over semantically discriminative visual objects across time. However, existing object-agnostic solutions struggle to handle substantial object variations over time. Current state-of-the-art approaches generally fall into two paradigms that exhibit critical limitations:

1. Vanilla RL-based methods: While they use reinforcement learning (RL) to refine decisions, they often operate under a single-step perception regime where visual encoding is performed once. This lacks dynamic, multi-step interaction with visual evidence.
2. Chain-of-Thought (CoT) based methods: These adopt step-by-step reasoning but remain fundamentally language-centric. They rely on static inference frameworks and lack explicit grounding in fine-grained visual evidence during the reasoning process.

Consequently, both paradigms tend to rely on superficial, visually prominent (saliency-driven) cues rather than anchoring predictions on subtle yet critical visual evidence. This leads to inconsistent reasoning, biased conclusions, and a failure to capture complex temporal and object-level dependencies.

Methodology: Chain-of-Glimpse

To address these limitations, the authors propose Chain-of-Glimpse, a search-guided progressive object-grounded reasoning framework. The core philosophy is to enable Multi-modal Large Language Models (MLLMs) to "think with videos" by actively seeking and verifying visual evidence across frames. The framework consists of two tightly coupled components:

1. Object-Grounded Video Reasoning Formulation (OGVRF)

This component casts video reasoning as a multi-step process over object-level representations rather than a one-shot mapping.

• State Definition: The reasoning state $h^{(k)}$ at step $k$ summarizes previously selected evidence and multi-modal context.
• Action Space: Actions are defined as the selection of specific object instances $(t_k, m_k)$ from detected objects in frame $t_k$.
• Search Mechanism: Instead of committing to a single autoregressive path, the model employs Monte Carlo Tree Search (MCTS) to explore multiple object-grounded reasoning trajectories. This allows the model to balance exploration and exploitation, mitigating early commitment to visually salient but task-irrelevant evidence.
• Temporal Constraints: A monotonic constraint ($t_{k+1} \ge t_k$) ensures reasoning progresses forward in time, though the search tree allows revisiting previously grounded objects via alternative branches.
• Supervised Initialization: High-quality trajectories identified by MCTS are distilled into the model via Supervised Fine-Tuning (SFT) to create a reference policy ($\pi_{ref}$).

2. Multi-turn Decision Policy Learning (MTDP)

This component optimizes the reasoning policy under sparse, task-level rewards to ensure stable multi-step reasoning.

• Algorithm: The authors utilize Group Relative Policy Optimization (GRPO). For each training sample, a group of object-grounded reasoning trajectories is sampled.
• Reward Design: A composite reward function is used to align evidence selection with task-level correctness rather than visual prominence:
  $$R = R_{ans} + \alpha \cdot \frac{1}{K} \sum R_{evid}$$
  Where $R_{ans}$ is the answer correctness reward, $R_{evid}$ is the object-level evidence quality reward, and $\alpha$ balances the two. This incentivizes the model to suppress salient but irrelevant cues and focus on task-relevant evidence.
• Optimization: The policy is updated using a clipped importance-weighted objective with KL regularization against the reference policy, encouraging the model to explore high-reward trajectories while maintaining stability.

Key Contributions

The paper outlines three primary contributions:

1. Object-Grounded Video Reasoning Formulation (OGVRF): A principled framework that models video reasoning as a multi-step process over object-level representations. It explicitly grounds task-relevant visual evidence across frames, mitigating over-reliance on saliency-driven cues.
2. Multi-turn Decision Policy (MTDP): A learning mechanism that formulates multi-step reasoning as a stochastic trajectory optimization problem. Through policy optimization with grounding-aware rewards, it enables MLLMs to iteratively explore and consolidate high-reward reasoning trajectories, supporting interpretable and self-correcting decisions.
3. Extensive Empirical Validation: Comprehensive evaluations across general (NExTQA, Video-Holmes, CG-Bench-Reasoning, VRBench) and grounded (NExT-GQA) video reasoning benchmarks demonstrate consistent performance gains and strong generalization.

Experimental Results

The authors evaluated Chain-of-Glimpse using Qwen2.5-VL backbones (3B and 7B parameters) against various baselines, including proprietary models (GPT-4o, OpenAI o4-mini) and open-source MLLMs.

• General Video Reasoning (NExTQA): Chain-of-Glimpse-7B achieved 83.3% accuracy, outperforming the baseline Qwen2.5-VL-7B (79.7%) and proprietary systems like VideoAgent (71.3%) and LLoVi (66.3%). Improvements were particularly notable in Causal (83.7% vs 79.8%) and Temporal (80.4% vs 77.6%) reasoning categories.
• Complex Reasoning Benchmarks: On Video-Holmes, CG-Bench-Reasoning, and VRBench, the method showed consistent improvements. For instance, on Video-Holmes, Chain-of-Glimpse-7B reached 42.5% average accuracy, surpassing the baseline Qwen2.5-VL-7B (38.8%) and approaching GPT-4o (42.0%) while outperforming it in specific categories like Intention & Motive Chaining (IMC) and Temporal Causal Inference (TCI).
• Ablation Studies:
  • Components: Removing MCTS, SFT, or GRPO individually resulted in performance drops, confirming the necessity of the full pipeline.
  • Search Depth: Moderate MCTS rollouts (e.g., 8) improved performance without significant computational overhead. Increasing rollouts beyond 8 yielded diminishing returns or increased latency.
  • Reward Design: A balanced reward ($\alpha=0.5$) between answer correctness and evidence quality yielded the best results; excessive focus on intermediate evidence degraded final accuracy.
  • Video Length: The method significantly improved performance on long videos (≥1,000 frames), reducing the gap between short and long video performance observed in baselines.

Significance and Claims

The paper claims that Chain-of-Glimpse successfully bridges the gap between low-level visual perception and high-level logical reasoning in video understanding. By explicitly anchoring each reasoning step to specific visual evidence regions, the framework enables compositional and multi-step decision-making that is both accurate and interpretable.

The authors emphasize that their approach moves beyond passive perception to active reasoning, emulating human-like object-grounded cognition. The results suggest that progressive object-grounded decision modeling substantially improves out-of-domain generalization and robustness across diverse video reasoning tasks, offering a viable path toward more reliable multi-modal AI systems without relying solely on massive proprietary models.