💬 NLP

The Abstraction Gap in Vision-Language Causal Reasoning

This paper introduces the CAGE benchmark and a dual-probe methodology to reveal a significant "abstraction gap" where most vision-language models generate linguistically plausible but causally unfaithful explanations, demonstrating that while the capability for faithful reasoning exists in certain architectures, it is not reliably achieved through standard fine-tuning.

Original authors: Chinh Hoang, Mohammad Rashedul Hasan

Published 2026-05-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chinh Hoang, Mohammad Rashedul Hasan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Talking the Talk vs. Walking the Walk

Imagine you are interviewing a student for a job. You ask them, "If it starts raining, what happens to the picnic?"

The student answers smoothly: "Well, if it rains, the picnic will get wet, the food will spoil, and everyone will run for cover. It's a total disaster!"

You are impressed. They sound smart, logical, and fluent. But then you ask them to draw a simple flowchart on a whiteboard showing why that happens. They freeze. They can't draw the arrows connecting "Rain" to "Wet Ground" to "Spoiled Food." They just know the words, but they don't actually understand the mechanics of the situation.

This paper argues that current AI models (Vision-Language Models or VLMs) are exactly like that student. They can generate beautiful, convincing stories about cause-and-effect in pictures, but when you ask them to show the actual "skeleton" of their reasoning, they often fail.

The Problem: The "Abstraction Gap"

The researchers call this disconnect the Abstraction Gap. It's the distance between:

Plausibility: How good the answer sounds (linguistic fluency).
Faithfulness: How accurately the answer reflects the model's actual internal reasoning (structural understanding).

The paper claims that for most AI models, the "sound good" score is high, but the "understand the structure" score is near zero.

The Experiment: CAGE (The "Causal Gym")

To test this, the authors built a new gym called CAGE (Causal Abstraction Gap Evaluation). They took 5,500 real-world photos (like a beach, a park, or a street) and asked the AI three types of questions based on a famous "Ladder of Causation":

Level 1 (Looking): "What color is the umbrella?" (Easy, just observation).
Level 2 (Changing): "If a strong wind blows from the mountains, what happens to the umbrellas?" (Hypothetical action).
Level 3 (Imagining): "If this photo was taken on a busy holiday instead of a quiet day, how would the scene look different?" (Counterfactual).

The Twist:
For the hard questions (Levels 2 and 3), the researchers forced the AI to do two things in order:

First: Write a "causal chain" using simple arrows (e.g., Strong Wind → Force on Umbrella → Umbrella Falls).
Second: Write the full sentence explanation.

The Results: The "Magic Trick" Fails

The results were shocking.

The Text-Only Test: When the AI was just asked to write a sentence, it scored very high (around 7 or 8 out of 10). It sounded like a genius.
The Chain-Text Test: When the AI was forced to draw the arrow diagram first, the scores crashed. Most models scored below 2.5 out of 10. Many just gave blank answers or nonsense.

The Metaphor:
Think of the AI as a parrot. The parrot has memorized the phrase "The wind blows the umbrella away" because it heard it a million times in its training data. It can say the phrase perfectly. But if you ask the parrot to explain the physics of how the wind pushes the umbrella, it has no idea. It's just mimicking the sound of reasoning, not doing the reasoning itself.

The "Magic" Model

Interestingly, one model (LLaVA-NeXT) managed to bridge this gap. It could write the arrow chain and the sentence with high scores. This proves that the technology can do it; it's just that most other models haven't learned how yet.

Why Can't We Just "Teach" Them?

The researchers tried to fix the problem by "fine-tuning" (re-training) the bad models on 45,000 examples that included the correct arrow chains. They hoped this would teach the models how to think structurally.

The Result: It didn't work.

The models got slightly better at writing sentences.
But they did not get better at drawing the arrow chains. In fact, for some models, forcing them to learn chains made them worse at everything.

The Analogy:
Imagine trying to teach a dog to play chess by showing it a picture of a chessboard and saying, "Move the pawn." The dog might learn to bark at the board (mimic the language), but it won't actually learn the rules of the game (the structure). You can't force a new type of thinking onto a model just by showing it more examples if the model's brain isn't built to handle that type of logic.

The "Two Brains" Discovery

The paper also found something weird about "hallucinations" (when AI makes things up that aren't in the picture).

Some models are great at not making things up about objects (e.g., they won't say there's a cat if there isn't one).
But these same models are terrible at understanding relationships (e.g., they can't figure out that wind causes umbrellas to fall).

It's like having a person with two separate brains:

Brain A is great at spotting objects (Is there a ball? Yes/No).
Brain B is terrible at understanding how things interact (If I kick the ball, it rolls).
Training Brain A doesn't help Brain B, and vice versa.

The Bottom Line

The paper concludes that fluent language is not proof of understanding. Just because an AI can write a convincing story about cause-and-effect doesn't mean it actually understands the world.

The Gap: There is a huge gap between sounding smart and being smart.
The Cause: Current AI training focuses too much on making the AI sound human (fluency) and not enough on making it build logical structures (reasoning).
The Future: To get truly reliable AI for important tasks, we need to stop just asking for "answers" and start demanding that the AI show its "work" (the causal chains) before it gives the answer. If it can't show the work, it probably doesn't know the answer.

Technical Summary: The Abstraction Gap in Vision-Language Causal Reasoning

Problem Statement

Vision-language models (VLMs) have demonstrated impressive capabilities in generating fluent, contextually appropriate causal explanations for visual scenarios. However, a critical uncertainty remains: do these models possess genuine structural understanding of causal mechanisms, or are they merely generating linguistically plausible text through statistical pattern matching? Current evaluation methods often fail to distinguish between plausibility (linguistic fluency that inspires human trust) and faithfulness (an accurate reflection of the model's internal reasoning process). This gap poses significant risks in high-stakes domains such as medical imaging and autonomous driving, where understanding the reasoning behind a decision is as consequential as the decision itself. While recent benchmarks have tested VLMs' ability to select valid causal chains from pre-provided candidates, they do not assess the model's ability to generate these structures independently.

Methodology

To address this, the authors introduce a dual-probe evaluation framework designed to isolate linguistic fluency from structural abstraction. This framework relies on Judea Pearl's Ladder of Causation, categorizing questions into three levels:

Level 1 (Association): Observing correlations.
Level 2 (Intervention): Reasoning about the effects of hypothetical actions.
Level 3 (Counterfactual): Imagining alternative scenarios contradicting observed reality.

The methodology employs two distinct probes:

Text-Only Probe: Measures plausibility by evaluating standard textual responses without structural constraints.
Chain-Text Probe: Measures faithfulness by requiring models to first generate an explicit, lightweight causal chain (e.g., $A \rightarrow B \rightarrow C$ ) before providing a textual explanation. This ordering mirrors the cognitive sequence of structure-before-expression, hypothesizing that genuine causal understanding requires constructing a structural representation before verbalizing it.

The authors define the Abstraction Gap (AG) metric to quantify the disparity between these two capabilities:
$AG_\ell(M) = \frac{E[T(M, d)] - E[C(M, d)]}{E[T(M, d)]}$
where $T$ and $C$ represent scores for text and chain generation, respectively. A high AG indicates that a model produces fluent text but fails to generate valid structural representations.

The evaluation is conducted on CAGE (Causal Abstraction Gap Evaluation), a new benchmark comprising 49,500 open-ended questions across 5,500 real-world COCO images. CAGE spans all three levels of Pearl's hierarchy and requires generation without scaffolding. The study evaluates eight open-source VLMs using a multi-judge framework (GPT-4o and Claude 3.5 Sonnet) with human validation to ensure reliability.

Key Contributions

CAGE Benchmark: The first large-scale visual causal reasoning benchmark requiring explicit structural output generation (causal chains) grounded in real-world images across Pearl's three causal levels.
Dual-Probe Evaluation Framework: A methodology that isolates abstraction capability from linguistic fluency, validated through multi-judge evaluation and human agreement studies.
Discovery of the Abstraction Gap: Systematic evaluation revealing a severe gap between linguistic fluency and structured reasoning in most current VLMs.
Training Framework Analysis: Experiments demonstrating that explicit chain supervision via fine-tuning on 45,000 chain-annotated examples fails to close the gap for models with low baseline performance, suggesting that structural abstraction capabilities are not easily instilled through standard fine-tuning if absent during pretraining.

Key Results

Severe Abstraction Gap: Seven out of eight evaluated VLMs exhibited an AG exceeding 0.50. These models achieved high text scores (6–8/10) but scored below 2.5/10 on chain generation. For instance, CogVLM2 achieved a Level 3 text score of 8.11 but only 1.37 for chain generation.
LLaVA-NeXT Exception: One model, LLaVA-NeXT, achieved a near-zero AG (0.03), demonstrating that the capability for faithful causal chain generation exists within current VLM architectures and depends on specific pretraining and architectural choices.
Fine-Tuning Limitations: Fine-tuning on 45,000 chain-annotated examples failed to close the gap for models with low baseline performance (e.g., MiniGPT-4 and mPLUG-Owl2). In some cases, fine-tuning degraded chain generation performance or increased blank outputs. Conversely, models with some initial structural capability (e.g., Qwen-VL-Chat) showed improvement, supporting the hypothesis that fine-tuning can activate existing capabilities but cannot create them from scratch.
Independence of Grounding: The study found that perceptual grounding (avoiding object hallucinations) and structural grounding (causal abstraction) are independent capabilities. Fine-tuning on causal chains did not affect hallucination rates, and models trained for hallucination mitigation (e.g., LLaVA-RLHF) still exhibited severe AG.
Recent Model Performance: Evaluation of recent models showed that while open-source models like Qwen3-VL still exhibit a moderate gap (AG = 0.313), the closed-source Gemini 2.5 Flash achieved a near-zero AG (-0.006), suggesting that architectural choices (such as sparse mixture-of-experts) may help preserve structural generation capabilities.

Significance and Claims

The paper claims that the Abstraction Gap exposes a fundamental structural deficit in current VLMs: the ability to generate linguistically plausible causal explanations does not entail the ability to represent causal structure. This finding provides empirical grounding for the distinction between plausibility and faithfulness in explainable AI.

The authors argue that the verification-generation asymmetry (where models perform well on chain selection but poorly on generation) indicates that standard multiple-choice evaluations may overestimate VLM reasoning capabilities. Furthermore, the independence of perceptual and structural grounding implies that interventions targeting one form of reliability (e.g., reducing hallucinations) should not be assumed to transfer to the other.

The study concludes that the Abstraction Gap is not an inevitable property of VLM architectures but is influenced by training procedures and alignment methods. Specifically, RLHF optimization for fluent text may suppress structured outputs in dense architectures, whereas architectures like MoE (Mixture of Experts) may better preserve the separation between linguistic processing and structural reasoning. The CAGE benchmark is released as a diagnostic tool to assess faithful causal reasoning before deploying VLMs in high-stakes applications.