💻 computer science

Semantic Richness or Geometric Reasoning? The Fragility of VLM's Visual Invariance

This paper reveals that despite their advanced semantic capabilities, state-of-the-art Vision-Language Models exhibit a fundamental fragility in geometric reasoning, failing to maintain spatial invariance and object identity under basic transformations like rotation and scaling, particularly when semantic content is sparse.

Original authors: Jason Qiu, Zachary Meurer, Xavier Thomas, Deepti Ghadiyaram

Published 2026-04-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jason Qiu, Zachary Meurer, Xavier Thomas, Deepti Ghadiyaram

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-traveled robot named VisionBot. VisionBot has read billions of books and seen millions of photos. It's great at describing what it sees: "That's a golden retriever," or "That's a red sports car." It knows the names and stories of things perfectly.

But this paper asks a very simple, almost childish question: If I turn that dog upside down, is it still the same dog?

The researchers found that while VisionBot is a genius at recognizing things in their "normal" positions, it completely falls apart when you rotate, shrink, or scribble them. It's like a person who can read a book in English but suddenly forgets how to read the same words if they are written upside down or in a different font.

Here is the breakdown of their discovery, using some everyday analogies:

1. The "Familiar Face" vs. The "Stranger"

The researchers tested VisionBot with two types of images:

The "Familiar Face" (Real Photos & Standard Text): Think of a photo of a cat or the letter "A" in a standard font. VisionBot is amazing here. It says, "Yes, that's a cat!" or "Yes, that's an A!" even if the cat is slightly turned.
The "Stranger" (Sketches & Weird Scripts): Now, imagine a simple stick-figure drawing of a cat, or a letter from an ancient language VisionBot has never seen (like Glagolitic). Suddenly, VisionBot gets confused. If you rotate that stick-figure cat, VisionBot might say, "No, that's not the same cat anymore!"

The Analogy: Imagine you are at a party. You easily recognize your best friend, even if they are wearing a hat or standing sideways. But if you see a stranger wearing a mask, and they turn around, you might not realize it's the same person. VisionBot is like that: it relies on memorizing the "face" (the semantic meaning) rather than understanding the geometry (the shape).

2. The "Magic Trick" of Rotation

The paper tested three specific tricks:

Rotation: Turning the image 90 or 180 degrees.
Scale: Making the image tiny or huge.
Identity: Showing two identical images to see if the bot knows they are the same.

The Result:

Real Photos: VisionBot passes with flying colors.
Sketches & Symbols: VisionBot fails miserably.
The "Rotation" Problem: This was the hardest. Even the smartest models (like Gemini-2.5-Pro) would look at a rotated letter "C" and say, "No, that's not a C," or "That's a different letter."

The Analogy: It's like a child who knows that a "cup" is a cup when it's sitting on a table. But if you turn the cup upside down, the child screams, "It's not a cup anymore! It's a bowl!" They haven't learned that the object stays the same even if its orientation changes.

3. Why Does This Happen? (The "Cheat Code" Theory)

The researchers discovered that VisionBot isn't actually doing "geometry." It isn't mentally rotating the shape in 3D space. Instead, it's cheating.

The Cheat: When it sees a photo of a dog, it recognizes the word "Dog" from its training data. It doesn't need to calculate angles; it just matches the pattern to a label it already knows.
The Failure: When the image is a weird sketch or a rotated symbol, the "label" is gone. The bot can't find the word "Dog" in its memory, so it panics and says, "I don't know this!"

The Analogy: Imagine a student taking a math test.

Scenario A: The teacher asks, "What is 2 + 2?" The student answers "4" instantly because they memorized the answer.
Scenario B: The teacher asks, "What is 2 + 2 if I write the numbers upside down?" The student freezes. They didn't actually learn how to add; they just memorized the specific look of the numbers.

VisionBot is that student. It memorized the "look" of things, not the "logic" of shapes.

4. Does Bigger Mean Better?

The researchers tested huge models (with billions of parameters) and smaller ones.

The Surprise: Making the model bigger didn't fix the problem. Even the "super-brains" failed at recognizing rotated sketches.
The Takeaway: You can't just feed a robot more data to fix this. It needs a fundamental change in how it "sees." It needs to learn geometry, not just memorization.

5. Why Should We Care?

This isn't just about failing a test. It's about safety.

Robotics: Imagine a robot doctor or a self-driving car. If a robot sees a "Stop" sign that is slightly tilted or covered in mud, and it fails to recognize it because it's not in the "perfect" position, that's dangerous.
The Future: We need to build robots that understand the shape of the world, not just the names of things. They need to know that a chair is still a chair, even if it's on its side.

Summary

The paper concludes that Vision-Language Models are "Semantic Rich" but "Geometric Poor."

They are like a person who knows the dictionary definitions of every word but has no idea how to draw a circle or understand that a square is still a square if you spin it. Until we teach them to understand the geometry of the world, they will remain fragile, failing whenever the world isn't perfectly aligned with their memory.

1. Problem Statement

The paper investigates a fundamental limitation in state-of-the-art Vision-Language Models (VLMs): their inability to perform robust geometric reasoning independent of semantic familiarity. While modern VLMs excel at semantic tasks (e.g., object recognition in natural scenes), they exhibit systematic failures when asked to determine object identity under basic geometric transformations (rotation, scaling, and identity matching) when semantic cues are sparse.

The core hypothesis is that current VLMs do not possess true geometric invariance/equivariance. Instead, their apparent success on natural images is a byproduct of dataset familiarity and reliance on semantic anchors (e.g., recognizing a "dog" or a Latin letter "A") rather than analyzing the underlying shape topology. When these semantic anchors are removed (e.g., using unfamiliar scripts or abstract sketches), performance collapses.

2. Methodology

Datasets and Visual Domains

The authors constructed a stress-test framework spanning a gradient of semantic richness:

Symbolic/Sparse:
- Omniglot: 1,623 handwritten characters from 50 diverse scripts (ranging from familiar Latin/Greek to rare Glagolitic/Braille).
- Times New Roman: Standardized English alphabet (high familiarity).
- Handwritten English: English alphabet with stroke variation.
Semantic/Rich:
- PACS: 800 images across 4 domains (Photo, Art, Cartoon, Sketch) for 7 object categories. This allows testing how performance degrades as texture and realism decrease.

Transformations

The study evaluates three fundamental transformations:

Rotation: Testing if $I'$ is a rotated version of $I$ (angles $10^\circ$ – $90^\circ$ for characters; $90^\circ, 180^\circ, 270^\circ$ for PACS).
Scale: Testing identity invariance across different resolutions ( $0.1\times$ to $0.9\times$ ).
Identity: Determining if two images depict the exact same character/object.

Models Evaluated

A diverse set of models was tested to ensure findings are not architecture-specific:

Closed-source: Gemini-2.5-Pro, GPT-5.2.
Open-source: Qwen2.5-VL (7B, 32B) and Qwen3-VL (8B, 30B).
Vision Encoders: CLIP, DINOv2, SigLIP, and Qwen's visual backbone were analyzed separately to isolate feature-level invariance from language-decoder reasoning.

Evaluation Metrics

Accuracy: Overall correctness.
True Positive Rate (TPR/Recall): Ability to correctly identify transformed variants as "Same."
True Negative Rate (TNR/Specificity): Ability to correctly identify different objects as "Different."
Prompting Strategies: Standard prompts, Object Recognition (OR), Object Identification (OI), and Rotation Recognition (RR) tasks were compared.

3. Key Contributions

Disentangling Semantics from Geometry: The paper provides empirical evidence that VLM "invariance" is largely a hallucination of semantic familiarity. Models perform near-perfectly on familiar scripts (Latin) and natural photos but fail catastrophically on unfamiliar scripts (Glagolitic) and abstract sketches, despite the geometric logic being identical.
The "Semantic Shortcut" Phenomenon: The authors demonstrate that models rely on object labels to bypass geometric reasoning. When asked "Is this a rotated 'C'?", models succeed. When asked "Is this a rotated [unfamiliar symbol]?", they fail, often defaulting to a bias of "No" (high TNR, near-zero TPR).
Vision Encoder vs. Language Decoder Disconnect: Analysis of feature cosine similarity (Fig. 2) shows that vision encoders (like SigLIP and Qwen's backbone) often maintain high similarity between rotated inputs. However, the language decoder fails to utilize this geometric signal, indicating the breakdown occurs in the multimodal fusion/reasoning stage.
Prompt and Capacity Insufficiency: The failure persists across model sizes (7B to 30B parameters) and is only partially mitigated by In-Context Learning (ICL) or structured visual prompts (Rotational Grids), which often induce confirmation bias (spiking TPR at the cost of TNR).

4. Key Results

Rotation Invariance Failure:
- On Times New Roman (familiar), Gemini-2.5-Pro achieved ~89% accuracy.
- On Omniglot (unfamiliar scripts), accuracy dropped to ~76% for the same model, with TPR plummeting to ~55%.
- On Sketches (PACS), even top models dropped significantly compared to Photos (e.g., Gemini-2.5-Pro: 92.67% on Photos vs. 73.17% on Sketches).
- Critical Finding: Open-source models (e.g., Qwen2.5-VL-32B) often achieved TPR < 15% on unfamiliar scripts, effectively guessing "No" for almost everything.
Scale and Identity:
- Models showed robustness to scale on familiar scripts (>98% accuracy) but degraded sharply on Omniglot (down to ~74-82%).
- Identity matching failed on unfamiliar scripts for smaller models (Qwen2.5-VL-7B had ~62% TPR on Omniglot identity tasks), proving a lack of geometric grounding.
Prompt Sensitivity:
- Models achieved near-perfect performance on Object Recognition (OR) and Object Identification (OI) tasks across all domains.
- Performance collapsed on Rotation Recognition (RR) tasks, confirming that the models recognize what an object is but cannot reason about how it transforms.
Mitigation Attempts:
- ICL (Few-shot): Improved TPR significantly (e.g., Qwen2.5-VL-32B on Malayalam went from 6% to 51%) but drastically reduced TNR (increased false positives), suggesting the models learned to guess "Yes" rather than reason geometrically.
- Rotational Grids: Similar pattern; improved TPR but caused severe TNR degradation, indicating the models became "over-eager" to confirm rotation without discriminative rigor.

5. Significance and Implications

Safety-Critical Risks: The inability to decouple semantic identity from geometric orientation poses severe risks for robotics and autonomous systems. A robot might fail to recognize a rotated obstacle or a manipulated tool if it relies solely on semantic priors.
Architectural Limitations: The findings suggest that current Transformer-based VLMs lack inherent equivariance mechanisms found in traditional CNNs. The "emergent" invariance observed in natural images is a data artifact, not a learned geometric principle.
Future Directions: The paper argues that future multimodal systems must move beyond surface-level data familiarity. It calls for:
- Architectural innovations that enforce geometric equivariance.
- Targeted data augmentations that force models to learn shape topology independent of semantic labels.
- Evaluation benchmarks that explicitly test zero-shot geometric reasoning on unfamiliar domains.

In conclusion, the paper establishes that current VLMs are semantically robust but geometrically fragile, relying on "semantic anchors" to navigate spatial tasks rather than possessing a fundamental understanding of geometry.