CaST-Bench: Benchmarking Causal Chain-Grounded Spatio-Temporal Reasoning for Video Question Answering

Technical Summary of CaST-Bench

Problem Statement
Vision-Language Models (VLMs) have advanced significantly in video understanding, yet they struggle with causal reasoning. Existing benchmarks primarily focus on descriptive questions (e.g., "what color is the bag?") or direct spatio-temporal localization. They rarely require models to identify a chain of linked evidences to explain why an event occurred. Furthermore, current datasets often lack fine-grained, grounded evidence (specific timestamps and bounding boxes) necessary to rigorously evaluate whether a model's reasoning is based on genuine visual causality or spurious correlations (confounders). This gap limits the development of transparent and trustworthy VLMs capable of deep causal analysis.

Methodology
To address this, the authors introduce CaST-Bench (Causal Chain-Grounded Spatio-Temporal Reasoning Benchmark) and a corresponding evaluation suite.

1. Dataset Construction (Human-AI Collaborative Pipeline):

   • Video Selection: The authors selected 1,015 complex, naturalistic videos from the SegmentAnything-Video (SAV) dataset, filtering for scenes with multiple interacting actors to ensure spatial ambiguity and causal complexity.
   • Instance Description: A pipeline generates dense, instance-level spatio-temporal descriptions. VLMs first generate descriptions for tracked instances, which are then validated and revised by human experts to ensure temporal accuracy and eliminate hallucinations. This resulted in 10,728 high-quality dynamic instance descriptions.
   • QA and Causal Chain Generation: A state-of-the-art VLM acts as a "causal thinker," generating structured Question-Answer (QA) pairs paired with explicit causal chains. The questions are categorized into four types: Causal Explanation, Counterfactual Reasoning, Predictive Anticipation, and Inferential Description.
   • Distractor Generation: To prevent models from relying on linguistic biases, the pipeline generates three types of distractors: text-based (plausible but incorrect based on language only), video-based (causally irrelevant visual instances), and "near-miss" options (subtle variations of the correct answer).
   • Filtering: A rigorous three-step filtering process ensures quality: (1) text-only filtering to remove questions solvable without video; (2) mask-based filtering where causal evidence regions are blacked out to verify that the question cannot be answered without the specific spatio-temporal evidence; and (3) human review. Only ~40% of initial pairs survive, resulting in 2,066 high-quality QA pairs.

2. Evaluation Suite:
   The authors propose a comprehensive evaluation framework that assesses not just answer correctness, but the quality of the reasoning process:

   • Grounded Causal Chain Metrics:
     • Instance-Matched tIoU (IM-tIoU): Measures temporal alignment between predicted and ground-truth evidence spans.
     • Instance-Matched vIoU (IM-vIoU): Measures joint spatio-temporal alignment (temporal overlap $\times$ spatial IoU of bounding boxes).
   • Answer Faithfulness Metrics:
     • Faithful Answer Rate (F): The rate of correct answers supported by well-grounded evidence.
     • Spurious Answer Rate (S): The rate of correct answers derived from poor or irrelevant grounding.
   • Open-Ended Evaluation: An LLM-as-a-judge framework evaluates logical consistency, evidence coverage, and overall justification quality.

Key Contributions

• Task Definition: The paper defines a new task: Causal Chain-Grounded Spatio-Temporal Reasoning, requiring models to actively search for and localize a chain of cause (prior evidence) and effect (outcome evidence) to answer questions.
• Benchmark (CaST-Bench): The first benchmark providing fine-grained spatio-temporal causal chains (timestamps and bounding boxes) for 2,066 questions across 1,015 videos, covering diverse real-world scenarios.
• Evaluation Suite: A novel set of metrics (IM-tIoU, IM-vIoU, Faithful/Spurious Rates) designed to evaluate the transparency and grounding of causal reasoning, distinguishing between correct answers derived from evidence versus those derived from biases.

Experimental Results
The authors evaluated 15 representative VLMs (including proprietary models like Gemini-2.5-Pro and GPT-5, and open-source models like Qwen3-VL and InternVL) against human baselines.

• Performance Gap: Current VLMs struggle significantly. Human annotators achieve 91.89% accuracy, while the best proprietary model (Gemini-2.5-Pro) scores 50.34%, and the best open-source model (Qwen3-VL-4B) scores 45.30%.
• Grounding Failure: Despite moderate answer accuracy, models perform poorly on grounding. IM-vIoU scores are extremely low (e.g., 2.46% for Gemini-2.5-Pro), indicating that models rarely identify the correct visual evidence.
• Faithfulness vs. Spuriousness: A high "Spurious Answer Rate" (30–45%) indicates that many correct answers are lucky guesses or based on irrelevant cues rather than causal chains.
• Ablation Studies: Providing ground-truth causal chains improves accuracy by 20–25 points, highlighting that the bottleneck is the construction of precise causal chains, not just the reasoning itself. "Thinking" variants of models do not consistently outperform "instruct" variants, suggesting that explicit textual reasoning traces do not automatically improve visual grounding.

Significance and Claims
The paper claims that CaST-Bench addresses a critical gap in video understanding by shifting the focus from surface-level perception to deep, grounded causal reasoning. The authors argue that:

1. Transparency: Enforcing visual evidence grounding makes VLMs more transparent and trustworthy, allowing users to distinguish between genuine understanding and reliance on spurious correlations.
2. Diagnostic Value: The benchmark serves as a rigorous diagnostic tool, revealing that current models fail primarily because they cannot construct precise, grounded causal chains, rather than a lack of general reasoning capability.
3. Future Direction: The results suggest that improving future VLMs requires a specific focus on the ability to locate, align, and chain spatio-temporal evidences, rather than simply scaling model size or adding generic "thinking" modules.

The authors maintain a modest tone, acknowledging limitations such as the scalability of their human-AI pipeline and potential biases inherited from the source SAV dataset, while emphasizing that the benchmark provides a necessary foundation for developing more robust and interpretable multimodal AI systems.