What's Missing in Vision-Language Models? Probing Their Struggles with Causal Order Reasoning

Technical Summary: What's Missing in Vision-Language Models? Probing Their Struggles with Causal Order Reasoning

Problem Statement
Despite the impressive performance of Vision-Language Models (VLMs) on downstream tasks such as visual question answering (VQA) and object detection, their ability to understand and reason about causal relationships in visual inputs remains unclear. Existing benchmarks often conflate causal reasoning with other reasoning types or allow models to exploit "shortcuts" based on object recognition and activity identification to arrive at correct answers without genuine causal understanding. Consequently, current evaluations fail to isolate and rigorously assess a model's specific capacity for causal reasoning. Furthermore, it is unknown whether VLMs can distinguish between valid and invalid causal directions (e.g., distinguishing "Rain causes the woman to hold an umbrella" from "Holding an umbrella causes rain").

Methodology
To address these gaps, the authors propose a comprehensive framework involving new benchmarks, controlled evaluation tasks, data-level analysis, and targeted fine-tuning strategies.

1. New Benchmarks (VQA-Causal and VCR-Causal):
   The authors introduce two new datasets specifically designed to isolate causal reasoning.

   • Construction: Derived from the validation sets of VQA and VCR, the datasets consist of 1,947 and 3,511 instances, respectively.
   • Format: Each image is paired with 12 pairs of captions. Within each pair, the captions are identical except for the causal order (e.g., "Event A is caused by Event B" vs. "Event B is caused by Event A").
   • Causal Conjunctions: The datasets utilize 12 distinct causal conjunctions (e.g., is caused by, because, result in, lead to) to test sensitivity to syntactic ordering variations.
   • Evaluation: Models are evaluated under a zero-shot setting. The metric measures the proportion of images where the model assigns a higher matching score to the correct causal caption than the incorrect one.

2. Controlled Evaluation (Object and Activity Understanding Test):
   To determine if poor causal performance stems from a lack of basic visual understanding, the authors constructed a modified test set. This set includes the same images as VQA-Causal but pairs them with captions describing objects and activities directly, rather than causal relationships. This isolates low-level feature recognition from high-level reasoning.

3. Data-Level Analysis:
   The authors investigated the prevalence of causal expressions in large-scale training corpora (LAION-400M, MSCOCO) and benchmark datasets (VQA, VCR). They quantified the frequency of explicit causal terms to explain potential training deficiencies.

4. Fine-Tuning Strategy (CausalCLIP):
   To mitigate the identified limitations, the authors developed a fine-tuning approach using "hard negative" cases.

   • Data: They extracted 4,891 "Why" questions from the VQA training set and generated caption pairs differing only in causal direction.
   • Training: Using a CLIP-based architecture (NegCLIP), they extended the contrastive learning objective. For each image, the model is trained to distinguish the correct causal caption from the incorrect causal order caption (hard negative) and other random negative captions.

Key Results

• Performance Gap: Across 12 evaluated VLMs (including CLIP, BLIP, LLaVA, and Qwen variants), performance on VQA-Causal and VCR-Causal was poor. Ten out of twelve models achieved accuracy no higher than 52%, which is only marginally above random guessing (50%) and significantly below human performance (~98%).
• Low-Level vs. High-Level: In the Object and Activity Understanding Test, models demonstrated strong performance (70–95% accuracy), confirming that the failure in causal reasoning is not due to an inability to recognize objects or activities, but rather a failure to capture high-level relationships between them.
• Data Scarcity: Analysis revealed that explicit causal expressions are extremely rare in training data: only ~0.08% in LAION-400M and ~0.01% in MSCOCO. Similarly, only 0.92% of VQA and 35.43% of VCR questions involve causal reasoning, with nearly half of the VCR causal questions solvable via object/activity shortcuts.
• Fine-Tuning Efficacy: The fine-tuned model, CausalCLIP, showed significant improvements. It achieved 61.46% accuracy on the in-domain VQA-Causal benchmark and 57.37% on the out-of-domain VCR-Causal benchmark. Crucially, this improvement was achieved without degrading downstream retrieval performance on MSCOCO and Flickr30K.
• Syntactic Sensitivity: Models exhibited varying sensitivity to causal conjunctions. For instance, CLIP variants performed better when the effect preceded the cause (e.g., is caused by) but struggled when the cause preceded the effect (e.g., result in).

Significance and Claims
The paper claims to provide the first rigorous, independent benchmarks for evaluating causal reasoning in VLMs, moving beyond the "shortcut" vulnerabilities of existing datasets. The authors argue that the primary limitation of current VLMs in causal reasoning is a data-level issue: the scarcity of explicit causal expressions in pre-training corpora prevents models from learning these relationships naturally.

The study demonstrates that while VLMs excel at low-level visual feature recognition, they lack robust causal reasoning capabilities. However, the authors show that targeted fine-tuning with hard negative examples can effectively bridge this gap, enhancing causal reasoning abilities while maintaining generalization and downstream performance. The work highlights a critical area for future research, suggesting that addressing causal understanding requires not just architectural changes but also a re-evaluation of training data composition.