Structured Causal Video Reasoning via Multi-Objective Alignment

1. Problem Statement

Current Video Large Language Models (Video-LLMs) struggle with complex video understanding tasks, particularly those requiring fine-grained temporal localization and causal inference. The core issues identified are:

• Unstructured Reasoning: Existing models often rely on unstructured Chain-of-Thought (CoT), which leads to verbose, non-linear narratives where critical visual cues are drowned in irrelevant tokens.
• Weak Causality & Temporal Drift: Models frequently fail to model temporal causality, reducing video analysis to isolated frame retrieval rather than understanding event dynamics. This results in "reasoning drift," where the model loses focus on the user query.
• Optimization Conflicts in RL: When applying Reinforcement Learning (RL) to improve reasoning, models face a multi-objective optimization problem. There is a natural trade-off between structural completeness (extracting dense facts), causal fidelity, and reasoning length (token efficiency). Standard RL algorithms (like GRPO) struggle to balance these competing objectives, often leading to training instability or suboptimal solutions where one objective (e.g., length) is sacrificed for another.

2. Methodology

The authors propose a Structure-First Paradigm and a novel training pipeline to bridge the gap between human cognitive processes (structured mental sketches) and model reasoning.

A. Structured Event Facts & Reasoning Pipeline

Instead of jumping directly to reasoning, the model is forced to generate a Structured Event Facts representation before reasoning. This acts as an explicit constraint.

• Structured Event Facts: A compact, high-density schema containing time, person, human action, scene, object, camera, and a causal event caption for specific temporal segments.
• Reasoning Process: The subsequent thinking phase is strictly constrained by these facts and follows three stages:
  1. Global Search & Localization: Scanning facts to identify candidate time windows.
  2. Causal Verification: Analyzing antecedents and consequences to verify temporal coherence and causal links.
  3. Final Alignment: Ensuring the inferred interval is fully contained within the observed activity.

B. Four-Stage Training Pipeline

To effectively train the model on this complex structure, the authors introduce a progressive pipeline:

1. Facts Training (Stage 1): Instruction tuning to generate accurate structured facts from videos.
2. Format Warm-Start (Stage 1.5): Training the model to output facts within the specific XML-like tags (<facts>, <thinking>, <answering>) to prevent structural hallucinations.
3. Thinking Warm-Start (Stage 2): Instruction tuning on causal reasoning tasks using the generated facts as context.
4. RL Post-Training (Stage 3): Reinforcement Learning to align the model with real-world problem-solving.

C. CausalFact-60K Dataset

A new dataset comprising 60,000 high-quality annotations constructed from Video Temporal Grounding (VTG) datasets (e.g., ActivityNet, Charades). It features:

• Dense Event Captions: Regenerated structured descriptions for every temporal segment.
• Causal Reasoning Traces: Explicit reasoning chains generated by strong LLMs (Gemini 2.5 Pro) conditioned on the facts.
• Quality Control: A rigorous pipeline involving automated quality judges and human inspection.

D. Pareto-Frontier guided Advantage Balancing (P-FAB)

To solve the multi-objective optimization challenge in the RL stage, the authors propose P-FAB, an algorithm inspired by the Multiple Gradient Descent Algorithm (MGDA).

• Problem: Standard Group Relative Policy Optimization (GRPO) averages rewards, which can obscure critical trade-offs and lead to ambiguous gradient directions when objectives conflict (e.g., maximizing fact density vs. minimizing token count).
• Solution: P-FAB treats each reward component (Format, Accuracy, Length, etc.) as an independent objective. It solves a minimum-norm problem in the standardized reward space to find a Pareto-optimal weight vector.
• Mechanism: It dynamically balances advantages by minimizing the norm of the weighted combination of centered rewards. This ensures that rare but critical signals (e.g., correct causal logic) are not overwhelmed by high-variance, easily satisfied objectives.

3. Key Contributions

1. Structured Event Facts: Introduction of a "Structure-First" approach where models generate a compact, causal schema before reasoning, significantly improving interpretability and temporal grounding.
2. CausalFact-60K: A large-scale, high-quality dataset specifically designed for training structured causal reasoning in videos.
3. P-FAB Algorithm: A novel Multi-Objective Reinforcement Learning (MORL) algorithm that resolves reward conflicts by optimizing toward the Pareto-Frontier, outperforming standard GRPO in balancing competing objectives.
4. Factum-4B: A 4B-parameter Video-LLM that demonstrates state-of-the-art performance, proving that smaller models can outperform larger ones if equipped with structured reasoning and proper alignment.

4. Experimental Results

The model, Factum-4B, was evaluated on multiple benchmarks:

• Temporal Grounding:
  • ActivityNet-Captions: Achieved R1@0.5 of 48.4% and R1@0.7 of 28.1%, significantly outperforming 7B-scale models like Time-R1-7B and VideoChat-R1-7B.
  • Charades-TimeLens: Achieved R1@0.7 of 21.6%, the highest among open-source models.
• General Video Understanding:
  • VideoMME: Achieved 64.7% accuracy, surpassing the base Qwen3-VL-4B-Thinking model.
  • NExT-GQA: Achieved 73.6% accuracy, a 7% absolute gain over the thinking base.
  • ETBench: Led in 6 out of 8 sub-tasks, including Temporal Video Grounding (TVG) and Temporal Event Matching (TEM).
• Ablation Studies:
  • Removing the "Thinking" component caused the most severe performance drop, confirming the necessity of the reasoning bridge.
  • Removing "Facts" also significantly degraded performance, highlighting the role of facts as anchors.
  • P-FAB vs. GRPO: P-FAB consistently outperformed standard GRPO, with the performance gap widening as the group size increased (e.g., from 4 to 8 samples), demonstrating its superior ability to navigate complex multi-objective spaces.

5. Significance

This work addresses a fundamental cognitive gap in Video-LLMs by moving away from unstructured, verbose reasoning toward evidence-grounded, structured causal inference.

• Efficiency: It demonstrates that a 4B model can outperform larger proprietary and open-source models by using a structured reasoning framework rather than brute-force scaling.
• Robustness: The P-FAB algorithm provides a generalizable solution for multi-objective alignment in RL, which is crucial for future complex AI systems where trade-offs between accuracy, speed, and structure are inevitable.
• Interpretability: By forcing the model to output structured facts and explicit causal verification steps, the reasoning process becomes transparent and verifiable, reducing hallucinations.

In summary, the paper establishes that structured priors combined with multi-objective optimization are key to unlocking reliable, fine-grained temporal and causal reasoning in video understanding.