🤖 AI

The Geometry of Representational Failures in Vision Language Models

This paper investigates the representational geometry of Vision-Language Models to reveal that the geometric overlap between latent "concept vectors" quantitatively explains specific visual failure patterns, such as hallucinations and object confusion, which can be reliably manipulated through steering interventions.

Original authors: Daniele Savietto, Declan Campbell, André Panisson, Marco Nurisso, Giovanni Petri, Jonathan D. Cohen, Alan Perotti

Published 2026-06-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Daniele Savietto, Declan Campbell, André Panisson, Marco Nurisso, Giovanni Petri, Jonathan D. Cohen, Alan Perotti

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are looking at a busy party through a pair of special glasses. You see a red circle, a blue square, and a green triangle. A human brain can easily keep these three distinct items separate in its mind. But for the Vision-Language Models (VLMs) studied in this paper, looking at that same party is like trying to hold three different conversations in your head at once while someone is shouting over you. Sometimes, the brain gets confused and tells you, "I see a red square!" even though there is no such thing. It has mixed up the red from the circle with the square.

This paper investigates why these AI models make these specific "mix-up" mistakes, which researchers call hallucinations or illusory conjunctions.

Here is the breakdown of their findings using simple analogies:

1. The Problem: The "Crowded Room" of Ideas

Think of the AI's brain as a giant, high-dimensional room where every concept (like "red," "blue," "square," "circle") has its own specific spot.

The Human Way: Humans use "serial attention." We look at the red circle, lock it in our memory, then look at the blue square. We process them one by one.
The AI Way: These models try to process the whole scene in one giant, instant snapshot. They have to fit all the concepts into that single room at the same time.

The paper argues that when too many concepts are in the room at once, their "spots" start to overlap. If the spot for "red" is too close to the spot for "square," the AI gets confused and blends them together. This is called Geometric Interference.

2. The Discovery: Mapping the "Concept Map"

The researchers wanted to see if they could find the actual "coordinates" of these concepts inside the AI's brain. They used two main tools:

The "Probe" (A Detective): They tried to train a simple detector to find "red" or "blue." However, this was like trying to find a needle in a haystack by just guessing; the detector often found shortcuts that didn't actually represent the true concept.
The "Centroid" (The Average): Instead of guessing, they took hundreds of images of red things, found the mathematical "center of gravity" for all of them, and used that as the true definition of "red."

The Result: The "Centroid" method worked much better. It found the true, stable location of "red" inside the AI's brain.

3. The Proof: The "Remote Control" Experiment

To prove they had found the real "red" button and not just a fake one, they performed a steering experiment.

The Analogy: Imagine you have a remote control that can change the color of a flower in a photo.
The Action: They took a picture of a red rose. They found the mathematical vector (the "button") for "red" and subtracted it. Then, they pressed the "blue" button.
The Outcome: The AI, which was originally describing a red rose, suddenly started describing a blue rose.
Why this matters: This proved that the researchers had found the actual "switch" inside the AI's brain that controls the concept of color. It wasn't just a coincidence; they had physically rewired the AI's perception for a moment.

4. The "Curse of Generalization"

The paper explains that this confusion isn't a bug; it's a side effect of the AI being too smart at generalizing.

The Analogy: Imagine you teach a child that "red" applies to apples, fire trucks, and strawberries. To do this, the child must create a broad, flexible category for "red."
The Trade-off: Because the AI has to make "red" flexible enough to cover many different shades and objects, the "red" concept becomes a bit "fuzzy" or "crowded." When the AI sees a red circle and a green square, the "red" part of the circle and the "square" part of the square get so close together in the AI's mind that they accidentally merge.
The Paper's Conclusion: The more flexible and generalizable the AI's understanding is, the more likely it is to get confused when multiple things are present at once. This is the "Curse of Generalization."

5. Predicting the Mistakes

The researchers found that they could predict exactly when the AI would fail just by measuring the distance between concepts in their "concept map."

If the "red" concept and the "square" concept are geometrically close to each other, the AI is very likely to mix them up.
If they are far apart, the AI gets it right.
They tested this on a "Visual Search" game (finding a specific item in a crowd). The AI failed more often when the "distractor" items were geometrically similar to the target item, exactly as their map predicted.

Summary

This paper shows that AI vision models don't just "see" things; they organize them in a geometric space. When they try to see too many things at once, the "shapes" of these ideas bump into each other, causing the AI to hallucinate. By mapping these ideas and even "steering" them like a remote control, the researchers proved that these errors are caused by the fundamental geometry of how the AI stores information, not just random glitches.

What the paper does NOT claim:

It does not claim this solves the problem for all AI.
It does not suggest using this to fix medical diagnoses or self-driving cars yet.
It does not say AI is "conscious" or "feels" like humans; it only says the mathematical structure of the errors looks similar to human cognitive limits.

Technical Summary: The Geometry of Representational Failures in Vision Language Models

1. Problem Statement

Vision-Language Models (VLMs) exhibit systematic failures in multi-object visual tasks, including miscounting objects, confusing attribute-object bindings (e.g., attributing the wrong color to a shape), and generating "illusory conjunctions." While these errors mirror human cognitive limitations such as the "Binding Problem," the internal mechanisms driving them in artificial systems remain poorly understood.

The paper posits that these failures are not merely behavioral artifacts but stem from geometric representational interference. In the high-dimensional latent space shared by VLMs, vectors encoding distinct concepts (e.g., "red" and "square") may overlap or clash when multiple objects are processed simultaneously. Unlike biological vision, which mitigates this through serial attention, current VLMs must resolve all concepts in a single feedforward pass, leading to unavoidable feature blending when representations are not sufficiently orthogonal.

2. Methodology

The authors propose a framework to extract, validate, and analyze the representational geometry of open-weight VLMs (specifically Qwen2.5-VL, InternVL2.5, and Gemma 3).

A. Concept Vector Extraction

The study compares two primary methods for distilling "concept vectors" ( $v_c$ ) that encode visual features (colors, shapes, and their conjunctions):

Probe-Based Method: Training linear classifiers (attention probes) to distinguish the presence or absence of a concept. The learned attention vector serves as the candidate concept vector.
Centroid-Based Method: Computing the centroid of token embeddings from images containing a specific concept, then projecting this centroid onto the subspace orthogonal to the global activation mean to isolate concept-specific signals.
Structural Regularization (PCA-Probe): To address the potential for probes to learn "discriminative shortcuts," the authors apply Principal Component Analysis (PCA) to probe vectors. This forces the vectors to align with the assumed factorial structure of the concepts (e.g., separating color and shape axes), acting as a geometric regularizer.

B. Causal Validation via Activation Steering

To verify that extracted vectors are causally active rather than mere correlations, the authors employ activation steering.

Mechanism: For an image containing concept $A$ , the component of the activation vector aligned with $A$ is subtracted, and a component aligned with concept $B$ is injected.
Intervention: $h'_t = h_t - (h_t^\top \hat{v}_A)\hat{v}_A + (h_t^\top \hat{v}_A)\hat{v}_B$ .
Metric: Success is measured by whether the model's output shifts from describing $A$ to describing $B$ (e.g., perceiving a red flower as blue) without disrupting unrelated visual information.

C. Geometric Analysis

The authors analyze the cosine similarity between concept vectors to map the representational manifold. They investigate:

Compositional Structure: Whether color-shape conjunctions factorize into independent color and shape axes.
Semantic Structure: The geometry of continuous color spaces (100 hues) to identify "accordion effects" (within-category compression and between-category separation).
Correlation with Errors: Linking the geometric overlap (cosine similarity) between target and distractor vectors to model accuracy in visual search and similarity estimation tasks.

3. Key Results

A. Extraction Method Efficacy

Centroid Superiority: Centroid-based extraction consistently outperforms naive probing in causal steering tasks. For natural image color steering, centroids achieved 84.7%–95.7% accuracy across models, whereas naive probes often failed (<4% on Qwen/Gemma).
Role of Regularization: PCA regularization improved probe performance significantly for some models (e.g., InternVL reached 92.0%) but remained inconsistent across architectures.
Compositional Binding: For color-shape conjunctions, centroid vectors achieved the highest steering success (78.1% for Qwen, 75.3% for Gemma), suggesting that the internal binding mechanism is additive and aligns with the arithmetic mean of activations.

B. Representational Geometry

Compositional Manifolds: Concept vectors for color-shape pairs form structured manifolds where similarity depends strictly on shared features. Objects sharing a color or shape cluster distinctly, confirming that color and shape are encoded along independent axes.
Semantic Warping: The continuous color space is not a faithful physical map but a semantic one warped by the LLM's discrete token space. It exhibits "categorical perception," where hues within a linguistic category (e.g., "green") are compressed, while boundaries between categories are expanded.
The "Mexican Hat" Profile: Semantic similarity profiles show a monotonic decay for local distances but a rise in similarity for distant hues (distal interference), indicating finite representational resolution and potential for non-local interference.
Universality: Representational Similarity Analysis (RSA) reveals nearly identical geometric structures across Qwen, InternVL, and Gemma ( $r > 0.93$ ), suggesting these constraints are fundamental to the VLM paradigm rather than architecture-specific artifacts.

C. Correlation with Behavioral Failures

Visual Search: Model accuracy in visual search tasks decreases monotonically as the cosine similarity between the target and distractor concept vectors increases. High geometric overlap predicts higher error rates (false negatives when the target is present; false positives/illusory conjunctions when absent).
Similarity Judgments: Model confidence (logit separation) in color similarity tasks is better predicted by the cosine similarity of concept vectors than by perceptual HSV distance. This indicates that model uncertainty is governed by representational geometry rather than perceptual distance.

4. Significance and Claims

The paper claims to provide a mechanistic insight into VLM failures, framing them as the mathematical cost of compressing rich visual information into reusable, generalizable representations.

The "Curse of Generalization": The authors argue that the very geometric structures enabling flexible generalization (e.g., associating "red" with a continuous spectrum of shades) introduce susceptibility to inter-concept interference. This tradeoff is particularly acute for parallel processing systems like VLMs, which lack the serial attention mechanisms of biological vision.
Predictive Power: The geometry of concept vectors is not merely descriptive but predictive. Cosine similarity between vectors correlates strongly with model confidence ( $r > 0.77$ ) and accuracy ( $|r| > 0.83$ ) on downstream multi-object tasks.
Causal Validity: The study demonstrates that concept vectors derived from synthetic data are causally active, capable of reliably manipulating model perception in natural images.
Cognitive Parallel: The findings draw a parallel between VLM failures and human "illusory conjunctions," suggesting that both systems face similar constraints when mapping continuous spectra to discrete categories without serial attention.

The authors conclude that understanding these geometric signatures offers a grounded quantitative framework for anticipating model failures, potentially informing the development of more robust systems and better calibration of model confidence in safety-critical applications.