Chain-of-Frames: Advancing Video Understanding in Multimodal LLMs via Frame-Aware Reasoning

1. Problem Statement

Current Multimodal Large Language Models (MLLMs) struggle with video understanding, particularly in tasks requiring temporal reasoning and causal inference. Existing approaches to integrating Chain-of-Thought (CoT) reasoning into video models face three primary limitations:

• Temporal Inconsistency: Methods that generate reasoning traces based on general video descriptions often lack precise alignment with specific video frames, leading to hallucinations or logical gaps regarding when events occur.
• Complex Inference Pipelines: State-of-the-art methods often rely on multi-stage architectures involving auxiliary models for key-frame selection, scene graph generation, or metadata extraction. This increases computational cost, latency, and reduces generalizability.
• Data Scarcity and Cost: Generating high-quality, frame-grounded reasoning traces for training is expensive. It typically requires iterative refinement by human annotators or complex LLM pipelines, limiting the scale of training datasets (e.g., previous datasets like VideoCoT are limited to ~11k samples).

2. Methodology: Chain-of-Frames (CoF)

The authors propose Chain-of-Frames (CoF), a unified, single-stage reasoning approach that explicitly integrates temporal grounding into the reasoning process without auxiliary modules.

A. Core Concept

Instead of generating generic reasoning, CoF requires the model to produce reasoning traces that explicitly reference specific frame identifiers (e.g., "Frame 5," "Frame 12") relevant to the query.

• Format: The reasoning trace is a natural language explanation where the model cites the frame number where a specific visual event or object appears.
• Advantage: This creates a direct link between the visual content and the logical steps, ensuring the model "looks" at the correct part of the video before concluding.

B. Data Generation Pipeline (COF-DATA)

To overcome data scarcity, the authors created COF-DATA, a large-scale dataset (164,186 samples) derived from both real and synthetic sources:

1. Real Videos (COF-DATA_real):
   • Source: VideoEspresso dataset.
   • Process: The authors align existing frame-level captions with frame IDs. They use an LLM (Llama-3.1-8B-Instruct) to generate question-reasoning-answer triplets based on these temporally aligned captions, ensuring the reasoning cites the correct frame numbers.
2. Synthetic Videos (COF-DATA_synth):
   • Source: CLEVRER dataset (3D object interactions).
   • Process: Leveraging the rich, ground-truth annotations of CLEVRER (object properties, velocity, location, collision events), the authors use manual templates to automatically generate questions, answers, and reasoning traces.
   • Key Insight: This allows for the generation of massive amounts of perfectly grounded data at near-zero cost, covering quantitative tasks like object counting, appearance order, and relative distance.

C. Model Training

The authors fine-tuned three open-source video LLMs (InternVL2.5-4B, InternVL3-8B, and Phi-3.5-Vision) on COF-DATA.

• Inference Strategy: The models are trained to output the reasoning trace (including frame references) followed by the final answer in a single forward pass.
• Input Handling: The models utilize the native video encoding format of InternVL (interleaving text identifiers like Frame-1 with image tokens), which naturally supports the CoF paradigm.

3. Key Contributions

1. Chain-of-Frames (CoF) Paradigm: A novel, single-stage reasoning framework that eliminates the need for complex multi-stage pipelines or auxiliary networks by embedding frame references directly into the reasoning trace.
2. COF-DATA Dataset: A large-scale, diverse dataset of 164k samples combining real-world and synthetic videos. The paper demonstrates that synthetic data alone (despite being out-of-distribution relative to real benchmarks) provides a significant performance boost, highlighting the efficacy of learning the task structure from synthetic data.
3. State-of-the-Art Performance: The CoF-enhanced models achieve superior results across multiple benchmarks, often outperforming much larger closed-source models (e.g., GPT-4o, Gemini-1.5) and other open-source SOTA models, despite having fewer parameters.

4. Experimental Results

The models were evaluated on five diverse benchmarks: VSI-Bench (spatial reasoning), Video-MME (comprehensive video understanding), MVBench (temporal dynamics), VidHal (hallucination), and EventHallusion (event hallucination).

• Performance Gains:
  • CoF-InternVL3-8B achieved an average accuracy improvement of +5.1% over the base InternVL3-8B.
  • CoF-InternVL2.5-4B improved by +3.8% over its base.
  • The CoF models outperformed the base models in nearly every benchmark, with significant gains in VSI-Bench (+10.3% for CoF-4B) and Video-MME (+3.4% for CoF-4B).
• Comparison to SOTA:
  • CoF-InternVL3-8B achieved the best results on VSI-Bench and MVBench, and the second-best on Video-MME and VidHal, surpassing larger models like Qwen2-VL-72B and LLaVA-OneVision-72B.
  • On EventHallusion, the CoF model achieved the best accuracy among all open-source models.
• Synthetic Data Impact: Models trained only on synthetic data (COF-DATA_synth) consistently outperformed those trained only on real data, proving that the model learns the underlying reasoning logic effectively from synthetic examples.
• Ablation Studies:
  • CoF outperformed standard CoT prompting and fine-tuning on QA pairs without reasoning.
  • Explicit frame references were crucial; removing them (SFT with CoT but no frame IDs) resulted in lower performance.

5. Significance and Impact

• Efficiency and Simplicity: CoF offers a "plug-and-play" solution for video reasoning. It removes the computational overhead of multi-stage inference frameworks, making advanced video reasoning accessible and efficient.
• Scalability: The demonstration that synthetic data can drive performance improvements in real-world video tasks suggests a scalable path for training future video models without relying on expensive human annotation.
• Interpretability: By forcing the model to cite specific frames, CoF provides a transparent reasoning path, allowing users to verify why a model made a decision (e.g., "The model identified the rescue in Frame 6"), thereby reducing temporal hallucinations.
• Generalization: The approach is model-agnostic (tested on InternVL and Phi-3.5) and task-agnostic, working effectively for both semantic questions (e.g., "What is the purpose?") and quantitative tasks (e.g., "How many collisions?").

In conclusion, Chain-of-Frames represents a significant step forward in video understanding by aligning the reasoning process with the temporal structure of the video, achieving state-of-the-art results through a simple, data-efficient, and interpretable framework.