💻 computer science

How Do VLMs Fail? Vision-Operation Misalignment in Compositional VQA

This paper introduces an operation-centric mechanistic framework to analyze Vision-Language Model failures in compositional VQA, revealing four distinct failure modes and demonstrating that they propagate through specific, dissociated computational pathways, thereby establishing a foundation for targeted reliability improvements.

Original authors: Navya Gupta, Bingjie Xu, Avinash Anand, Timothy Liu, Zhengchen Zhang

Published 2026-07-20

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Navya Gupta, Bingjie Xu, Avinash Anand, Timothy Liu, Zhengchen Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Detective, the Map, and the Glitchy Brain

Imagine you are trying to teach a robot to be a detective. You give it a picture of a messy room and a question like, "Is the red cat sitting on the blue chair?" To answer this, the robot can't just guess; it has to perform a series of mental steps. First, it has to find the cat (object selection). Then, it has to check if the cat is red (attribute verification). Finally, it has to look at the chair and figure out the spatial relationship between them (reasoning). This field of study is called Vision-Language Modeling, where computers try to understand both images and text at the same time.

For a long time, these robots were getting really good at answering questions overall. But there was a catch: they were sometimes "cheating." Instead of actually looking at the picture, they might just guess the answer based on how words usually go together in human language. For example, if asked, "Is the sky usually blue?" the robot might say "yes" without even looking at the image. Scientists wanted to know: when these robots get it wrong, why do they fail? Is it because they can't see the object? Is it because they can't figure out the relationship? Or are they just ignoring the picture entirely? Understanding this is crucial because if we want these robots to be reliable helpers in the real world, we need to know exactly which part of their brain is glitching so we can fix it.

The Operation-Centric Autopsy

In this paper, the researchers decided to stop just looking at the final score and start performing a "mechanistic autopsy" on the robot's brain. They used a specific dataset called GQA, which contains questions that require chaining multiple steps together, like a recipe. They focused on a model called Qwen2.5-VL-3B, which is a large language model with a vision component. Instead of just asking, "Did it get it right?", they asked, "How did it get there, and where did the signal break?"

They developed a clever way to test the robot's internal wiring. Imagine the robot's brain as a factory with two main assembly lines: one is the Attention Line, where the robot's "eyes" scan the image and decide which parts to focus on, and the other is the MLP Line (Feedforward Network), where the robot processes and transforms that information into a final thought. The researchers used "causal interventions," which is a fancy way of saying they temporarily broke specific parts of the robot's brain to see what happened. They would block the robot's ability to look at a specific object, or stop it from processing that object's details, and then see if the answer changed.

The Four Ways Robots Fail

By breaking the robot's brain in very specific ways, the authors discovered that there aren't just "right" and "wrong" answers. There are actually four distinct ways the robot can fail, and each failure happens in a different part of the factory:

The "Blind Spot" Failure (Grounding Failure):
This happens when the robot simply cannot find the object in the picture. It's like trying to find a specific red car in a parking lot, but the robot's eyes are closed. The researchers found that when the robot gets this wrong, it's because it never actually locked onto the visual details of the object. This failure is handled by the MLP Line (the processing factory). If you block the MLP from processing the object, the robot fails completely. It suggests the robot is struggling to turn the raw image data into a recognizable "object" in its mind.
The "Over-Focus" Failure (Reasoning Failure):
This is the most interesting and counter-intuitive one. Here, the robot does see the object perfectly, but it gets stuck staring at it. Imagine the robot sees a cat and a chair, but instead of thinking, "The cat is on the chair," it just obsesses over the cat's fur texture and forgets to compare it to the chair. The robot is "over-grounded." The study found that this specific failure travels through the Attention Line (the scanning mechanism), specifically in the later layers of the brain. The robot is looking at the right thing but failing to use that look to make a logical connection.
The "Wrong Property" Failure (Attribute Extraction Failure):
In this scenario, the robot finds the object and knows exactly where it is, but it gets the details wrong. It sees a golden cart but calls it silver. The robot has the visual signal, but the final step of converting that signal into a word fails. This, too, happens in the MLP Line, but specifically at the very end where the answer is generated. It's like the factory has the right raw material but the machine stamping the final label is broken.
The "Cheater" Failure (Language Prior Dominance):
Sometimes, the robot doesn't even look at the picture. If the question is "Is the chain black or yellow?" and the robot knows from its training that chains are usually black, it just guesses "black" without checking the image. The researchers found that for these types of questions, the robot's visual brain is barely active at all. It's relying entirely on its "language brain" to guess the answer. This is a complete bypass of the visual system.

The Great Pathway Split

The most significant discovery in the paper is a "pathway dissociation." The researchers showed that different types of errors travel through different wires in the robot's brain.

If the robot fails to find the object (Grounding) or fails to describe it correctly (Attribute Extraction), the problem is in the MLP processing line.
If the robot fails to reason about the relationship between objects (Reasoning), the problem is in the Attention scanning line.

This is a big deal because it means you can't fix all robot errors with one tool. If you try to fix a "reasoning" error by tweaking the object-processing line, you won't succeed. You have to target the specific wire where the error is happening.

Does This Happen in All Robots?

The researchers also checked if this was true for a different robot, LLaVA-1.5. They found that the "Cheater" and "Wrong Property" failures happened in both robots, suggesting these are fundamental issues with how the robot generates answers. However, the "Blind Spot" and "Over-Focus" failures only appeared in the first robot (Qwen2.5-VL). Why? Because the first robot has a vision encoder that keeps very detailed, pixel-perfect information about where objects are. The second robot (LLaVA) uses a vision encoder that summarizes the image more broadly, losing some of that fine spatial detail. Because the second robot doesn't have the same level of detailed "map," it doesn't suffer from the same specific types of "Blind Spot" or "Over-Focus" errors. This suggests that some errors only happen when the robot has a high-resolution map to get lost in.

The Bottom Line

The paper concludes that we can no longer treat Vision-Language Models as a single, black box that either works or doesn't. They are complex machines with different assembly lines for different jobs. When they fail, they fail in specific, predictable ways: sometimes they can't see, sometimes they can't think, and sometimes they just guess. By identifying exactly which "wire" is broken for each type of question, the authors provide a roadmap for building better, more reliable robots that actually look at the picture before they speak.

Technical Summary: How Do VLMs Fail? Vision-Operation Misalignment in Compositional VQA

Problem Statement

Compositional Visual Question Answering (VQA) requires Vision-Language Models (VLMs) to execute chained reasoning operations—such as object selection, spatial relation resolution, and attribute verification—over grounded visual evidence. While modern VLMs report strong aggregate accuracy on benchmarks like GQA, this performance masks systematic failure patterns. Models often succeed via linguistic shortcuts on easy operations while failing on tasks requiring genuine visual grounding and multi-step inference.

Existing analyses largely focus on broad failure types (e.g., object hallucination) or cross-modal information flow but lack a mechanistic understanding of how and where failures occur. Specifically, it remains unclear whether different compositional operations fail through distinct internal computational pathways (e.g., attention mechanisms vs. feedforward networks) and how these failures relate to the specific reasoning operations being performed.

Methodology

The authors propose an Operation-centric mechanistic framework to decompose VLM failures based on the specific reasoning operation and the internal computational pathway involved. The study utilizes the GQA dataset, which provides functional program annotations and scene graphs, allowing the authors to identify specific vision tokens causally relevant to each reasoning step rather than ablating all visual tokens uniformly.

The methodology employs three complementary causal interventions applied across all 36 layers of the Qwen2.5-VL-3B-Instruct model:

Causal Mean Ablation: Replaces the hidden states of specific grounding tokens (associated with a target object) with a neutral baseline (mean activation of all vision tokens). This measures the total causal dependence on the object's visual representation.
Attention Knockout: Sets attention weights from the answer token to the grounding object tokens to zero at specific layers, isolating the cross-token attention pathway.
MLP Knockout: Zeros out the output of the Feed-Forward Network (MLP) at specific token positions (either the object position or the answer position), isolating the within-token feedforward pathway.

The authors define Grounding Strength as the peak degradation in log-probability of the correct answer under mean ablation. They further compute the Grounding Specificity Shift (GSS) (Cohen's d) to quantify the difference in degradation between correct and incorrect predictions, which serves as the primary metric for classifying failure modes.

Key Contributions

1. Operation-Aware Failure Taxonomy

The paper identifies four mechanistically distinct failure modes, each characterized by a unique relationship between visual grounding strength and answer correctness:

Grounding Failure: Occurs when correct answers show substantially higher grounding strength than incorrect ones. The model fails because it never develops causal reliance on the target object's visual region (e.g., the select operation).
Reasoning Failure: Occurs when the relationship reverses; incorrect answers exhibit higher grounding strength than correct ones. The model locates the object but over-grounds on its visual representation during stages where abstract relational inference should dominate (e.g., the relate operation).
Attribute Extraction Failure: Occurs when grounding strength is high regardless of correctness. The model successfully grounds the object, but the final computation converting the grounded representation into an answer fails (e.g., verify, query, exist).
Language Prior Dominance Failure: Occurs when grounding strength is low for both correct and incorrect answers. The model relies on linguistic patterns and world knowledge rather than visual computation (e.g., choose, filter).

2. Pathway Dissociation

A central finding is the dissociation of computational pathways across failure modes:

Grounding and Attribute Extraction Failures route primarily through the Feedforward Network (MLP). For select, the failure lies in the object's representation processing; for verify and query, the failure lies in the answer-position MLP's ability to extract the correct attribute from a well-grounded representation.
Reasoning Failures route primarily through Late-Layer Attention. For relate, errors are driven by the answer token attending to the object's visual tokens in late layers (28–35), indicating an over-reliance on visual content rather than relational abstraction.

3. Cross-Architecture and Cross-Benchmark Validation

VSR Benchmark: Validation on the Visual Spatial Reasoning (VSR) benchmark confirms that spatial reasoning errors are MLP-mediated at the object encoding stage (specifically in bounding box token processing), not at the answer decoding stage. This distinguishes spatial reasoning failures from attribute extraction failures.
LLaVA-1.5-7B Analysis: When applied to LLaVA-1.5 (which uses a frozen CLIP encoder), Attribute Extraction and Language Prior failure modes remain consistent (architecture-independent). However, Grounding and Reasoning failures disappear. This suggests these modes are encoder-dependent, emerging only when the vision encoder preserves sufficient spatial granularity for the language model to fail at processing it.

Results

Grounding Failure (select): Shows a strong positive association between grounding strength and correctness ( $d = +0.75$ ). The failure is localized to the answer-position MLP, indicating that object recognition relies on within-token transformations rather than attention routing.
Reasoning Failure (relate): Shows a negative association ( $d = -0.17$ ), where incorrect answers are more dependent on the object's visual tokens. Attention knockout amplifies this effect, with errors concentrated in late layers, confirming that the model "over-grounds" on visual content when it should be reasoning abstractly.
Attribute Extraction (verify, query, exist): Shows high absolute grounding strength but no significant correctness discrimination in attention knockout. Failures are localized to the MLP at the answer position (for verify) or object positions (for exist), indicating a breakdown in converting grounded representations to specific attributes.
Language Prior (choose, filter): Shows negligible grounding strength and no significant pathway-specific effects, confirming these operations bypass visual computation.

Significance

The paper claims to provide a principled foundation for targeted improvements to VLM reliability in multimedia reasoning. By demonstrating that different failure types require fundamentally different corrective strategies, the work argues against one-size-fits-all solutions. Specifically:

Interventions targeting feedforward processing are necessary for grounding and attribute extraction errors.
Interventions targeting attention routing (particularly in late layers) are necessary for reasoning errors.
The distinction between task-level (architecture-independent) and encoder-dependent failure modes offers a diagnostic framework for understanding model limitations based on their visual encoding capabilities.

The authors emphasize that their contribution is diagnostic, offering a mechanistic map of how VLMs fail, rather than proposing a specific corrective intervention. This granular understanding is presented as a prerequisite for developing more robust and reliable VLMs.