Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts

This paper demonstrates that Vision-Language Models do not suffer from perceptual blindness when failing to ground visual evidence, but rather experience an arbitration failure where strong visual encoding exists yet is overridden by prior knowledge, a gap that can be addressed through targeted early-layer interventions.

The Gist

The Big Question: Are the Models Blind or Just Stubborn?

Imagine you show a robot a picture of a blue banana. You ask, "What color is this?"
The robot looks at the picture, processes the image, and then answers: "Yellow."

For a long time, researchers thought the robot was blind. They believed the robot's "eyes" (the vision part) failed to see the blue color, so it just guessed based on what it knew about bananas (that they are usually yellow).

This paper proves that theory wrong.

The authors found that the robot isn't blind. It sees the blue perfectly fine. The problem is that once it sees the blue, its "brain" (the language part) gets too stubborn. It hears a loud voice in its head saying, "Bananas are yellow!" and it ignores the blue eyes it just used.

The paper calls this "Arbitration Failure." The robot isn't failing to see; it's failing to decide what to say.

---

The Detective Work: How They Found Out

The researchers acted like detectives, using a special toolkit to peek inside the robot's brain layer by layer. Here is how they solved the case:

1. The "Logit Lens" (Peeking at the Thoughts)

Imagine the robot's brain is a long hallway with 30 rooms (layers). In each room, the robot whispers its current best guess.

• Early rooms: The robot whispers, "I see blue!" (It's looking at the picture).
• Middle rooms: It starts whispering, "But bananas are yellow..." (It's remembering facts).
• The "Crossover" Point: At a specific room, the "Yellow" whisper gets louder than the "Blue" whisper. This is where the robot decides to ignore the picture.

The researchers found that every robot sees the blue clearly in the early rooms. The visual signal is strong and clear. The failure happens later, when the robot decides to listen to its memory instead of its eyes.

2. The "Switch" Test (Proving Causality)

To be 100% sure, they did a "brain swap" experiment.

• They took a robot looking at a blue banana (which usually says "Yellow").
• They grabbed the "thoughts" (hidden states) from a robot looking at a real blue banana (which says "Blue").
• They swapped those thoughts into the first robot's brain at the critical "Crossover" room.

The Result: The robot suddenly changed its answer from "Yellow" to "Blue."
This proved that the information was there all along; it just needed a nudge to let the visual evidence win the argument.

Crucial Discovery: They tried swapping just the last thought (like we do with text-only AI), and it did nothing. Why? Because in these robots, the "blue" information is spread out across hundreds of tiny image-pixels, not just one spot. You have to swap the whole sequence of thoughts to fix it.

---

The Solution: Steering the Ship

If the robot sees the truth but gets bullied by its own memories, how do we fix it without re-teaching the whole robot (which is expensive and slow)?

The authors tried "Activation Steering." Think of this like a rudder on a ship.

• The Problem: The ship (the robot) is drifting toward the "Yellow" island because of a strong current (linguistic bias).
• The Fix: Instead of rebuilding the ship, they applied a tiny, precise push to the rudder early in the journey (in the early layers of the brain).
• The Result: This tiny nudge helped the ship stay on course toward the "Blue" island.

They found two ways to do this:

1. Linear Steering: A simple push in the right direction.
2. SAE Steering: A more sophisticated push that targets specific "features" of the thought process.

The Outcome: These free, training-free tweaks improved the robot's accuracy by up to 3.8%. It's not a magic cure-all, but it proves that we can fix the "stubbornness" without retraining the whole model.

---

The Takeaway: The "See vs. Act" Gap

The paper concludes with a powerful message for anyone building or using AI:

"The models already see well. The challenge is making them act on what they see."

The Analogy:
Imagine a person who is a brilliant art critic. They can look at a painting and perfectly describe the colors, the brushstrokes, and the lighting. But, if you ask them, "What is the main color?" and they have a strong habit of saying "Blue" because they love blue, they might ignore the painting and just say "Blue" anyway.

They aren't blind. They just have a bad habit of prioritizing their old opinions over new evidence.

What this means for the future:
We don't need to build better cameras (vision encoders) for these AI models. We need to build better judges (arbitration mechanisms) that know when to trust the eyes and when to ignore the old memories. The tools to do this (steering) already exist; we just need to use them.

Technical Summary

1. Problem Statement

Vision-Language Models (VLMs) often fail when visual evidence contradicts strong linguistic priors (e.g., identifying a blue banana as yellow). The prevailing hypothesis for this failure is perceptual blindness: the theory that the vision encoder fails to capture the conflicting visual detail, leaving the language model to rely solely on its prior knowledge.

The authors challenge this view, proposing instead that the failure lies in arbitration: the model correctly perceives and encodes the visual information, but the downstream decision-making mechanism (the "arbitration" layer) overrides the visual signal in favor of the linguistic prior.

2. Methodology

The study analyzes 10 VLMs (ranging from 7B to 72B parameters) across four architecture families (LLaVA, Qwen2-VL, InternVL, BLIP-2) using the Visual-Counterfact dataset (images with altered attributes like color or size). The investigation proceeds in four stages:

A. Multimodal Arbitration Crossover (MAC) Analysis

• Technique: Uses Logit Lens probing to track the vocabulary-level logits of competing answers (visual vs. prior) at every transformer layer.
• Protocol: Instead of matching single tokens, the authors use a six-variant token-matching protocol (lowercase, capitalized, uppercase, space-prefixed, hex subwords) to robustly identify the "visual" and "prior" tokens.
• Definition: The MAC layer is defined as the first layer where the visual logit stably exceeds the prior logit and persists.

B. Encoding–Grounding Dissociation

• Goal: To determine if grounding failures are due to weak encoding (perceptual blindness) or weak decision-making (arbitration).
• Metric: Measures the L2 distance between hidden states of counterfactual images (e.g., blue banana) and standard images (yellow banana) at various depths.
• Linear Probing: Trains logistic regression probes to classify visual attributes from hidden states to test linear separability.

C. Causal Validation via Activation Patching

• Technique: Full-sequence activation patching. Unlike standard LLM interpretability which patches only the last token, this method injects hidden states from a "standard" run into a "counterfactual" run across all token positions at the MAC layer.
• Hypothesis: If visual information is distributed across image tokens, last-token patching will fail, while full-sequence patching will flip the output.
• Decomposition: Separates patching effects by token type (image tokens vs. text tokens).

D. Inference-Time Intervention (Steering)

• Goal: To fix grounding errors without fine-tuning.
• Methods:
  1. Linear Steering: Adds a contrastive direction vector ($d = h_{std} - h_{cf}$) to hidden states.
  2. SAE-Guided Steering: Uses a Sparse Autoencoder (SAE) to identify and manipulate specific monosemantic features associated with visual vs. prior signals.
• Strategy: Applies steering to early layers (before the arbitration regime forms) using a residual application strategy to avoid information loss.

3. Key Contributions & Results

A. The Encoding–Grounding Dissociation

• Finding: Models do encode visual information correctly, even when they answer incorrectly.
• Evidence:
  • L2 Distance: The internal representation difference between a blue and yellow banana is statistically identical for models that answer correctly vs. incorrectly (Ratio $\approx 1.0$).
  • Linear Probes: Visual attributes are linearly decodable from early layers (10% depth) with AUC > 0.86 for both successful and failed samples.
  • Conclusion: The bottleneck is not perception; the visual signal is present but ignored during generation.

B. The Arbitration Mechanism

• MAC Layer Dynamics: The "crossover" point where visual signals overtake priors varies by architecture (36% to 71% of layers) and attribute type.
• Predictor of Success: The final-layer logit gap (difference between visual and prior logits) strongly predicts success ($\rho = 0.847$), whereas encoding strength does not ($\rho = 0.198$).
• Scaling: Larger models (e.g., 72B) commit to visual answers earlier and with larger logit margins, but the dissociation persists even at scale.

C. Causal Validation (Patching)

• Last-Token Failure: Standard last-token patching fails in VLMs (0–1% flip rate) because visual information is distributed across the sequence.
• Full-Sequence Success: Replacing hidden states across all tokens at the MAC layer flips 60–84% of outputs from the prior answer to the visual answer.
• Token Decomposition: Image tokens carry almost all causal impact; text tokens have negligible effect. At larger scales, image-only patching recovers 100% of the causal effect.

D. Intervention Success

• Training-Free Fixes: Applying linear or SAE-guided steering to early layers (not the MAC layer) improves visual grounding accuracy by +1.4% to +3.8%.
• Precision: SAE-guided steering shows higher precision (fewer degraded samples) than linear steering, particularly in models with complex arbitration dynamics.
• Key Insight: Interventions must happen before the arbitration decision is made (early layers), not at the point where the decision is observed (MAC layer).

4. Significance

• Paradigm Shift: The paper refutes the "perceptual blindness" hypothesis, establishing that VLMs suffer from arbitration failure. They see the truth but choose the prior.
• Methodological Advancement:
  • Introduces Full-Sequence Patching as a necessary tool for VLM interpretability, correcting the misconception that last-token interventions suffice.
  • Demonstrates that Logit Lens analysis must account for multi-variant token matching to accurately track visual signals.
• Practical Application: Provides a training-free, inference-time solution (activation steering) to mitigate hallucinations and improve grounding, offering a pathway to safer, more reliable VLMs without retraining.
• Architectural Insight: Highlights that the "connector" between vision and language in current VLMs is the site of the bottleneck, suggesting future architectures should focus on improving the arbitration mechanism rather than just the vision encoder.

5. Limitations

• Relies on synthetic counterfactual images, which may not capture the full complexity of natural visual-linguistic conflicts.
• Size-based reasoning showed sensitivity to paraphrasing, suggesting some keyword-matching behavior.
• Steering experiments were limited to 7–8B models; scaling these interventions to larger models is future work.

In summary, the paper concludes that VLMs already "see" well; the challenge is making them "act" on what they see. Targeted, training-free interventions in early layers can effectively bridge this gap.