💻 computer science

BareBones: Benchmarking Zero-Shot Geometric Comprehension in VLMs

The paper introduces "BareBones," a zero-shot benchmark using pixel-level silhouettes to demonstrate that state-of-the-art Vision-Language Models suffer a severe "Texture Bias Cliff" when deprived of RGB cues, revealing their reliance on texture and context rather than genuine geometric comprehension.

Original authors: Aaditya Baranwal, Vishal Yadav, Abhishek Rajora

Published 2026-04-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Aaditya Baranwal, Vishal Yadav, Abhishek Rajora

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are playing a game of "Guess the Animal" with a very smart, well-read friend.

The Game:
You show your friend a photo of a tiger. They instantly say, "Tiger!" because they see the orange fur, the black stripes, and the jungle background. Easy peasy.

The Twist:
Now, you take that same photo and turn it into a black-and-white silhouette. You erase the orange fur, the stripes, the background, and the lighting. All that's left is the outline of the animal's shape.

The Result:
Your friend, who was so confident before, suddenly freezes. They might guess "Cat," "Dog," or even "A giant blob." They can't tell it's a tiger anymore.

This is exactly what the paper "BareBones" discovered about modern Artificial Intelligence (specifically Vision-Language Models like GPT-4, Gemini, and LLaVA).

The Big Idea: The "Texture Bias Cliff"

The researchers built a new test called BareBones. They took thousands of images from different datasets (including a huge collection of Pokémon silhouettes) and stripped away everything except the shape.

They found something shocking: AI models are terrible at recognizing shapes if they can't see textures.

Here is the breakdown using simple analogies:

1. The "Cheat Code" Problem

Think of AI models like students who are great at memorizing answers but bad at understanding the concepts.

The Old Way (Texture): When an AI sees a tiger, it doesn't really "see" the shape of the tiger. It sees the stripes. It's like a student who knows the answer to a math problem only because they memorized the specific numbers on the page, not the formula.
The BareBones Way (Shape): When you remove the stripes (the texture) and leave only the outline, the AI loses its cheat code. It's like taking away the numbers and asking the student to solve the problem using only the formula. They fail miserably.

The authors call this the "Texture Bias Cliff." As soon as you remove the texture, the AI's performance doesn't just dip a little; it falls off a cliff.

2. The Pokémon Puzzle

To prove this wasn't just a fluke, the researchers used Pokémon.

The Challenge: Pokémon are famous for having very similar body shapes. A "Plusle" and a "Minun" look almost identical in outline, differing only in tiny details like ear tips or tail shapes.
The Test: They showed AI models 1,160 different Pokémon, but only as black-and-white outlines.
The Result: Even the smartest AI models (like GPT-4.1) got it wrong almost 100% of the time. They couldn't tell the difference between a "Gengar" and a "Mimikyu" just by looking at the outline.
The Human Comparison: A human Pokémon fan could guess about 70% of them correctly just by looking at the shapes. The AI, which is supposed to be "super smart," scored less than 4%.

3. Bigger Brains Don't Fix the Problem

You might think, "Maybe the AI just needs to be bigger or smarter?"
The researchers tested models ranging from tiny ones (1 billion parameters) to massive ones (26 billion parameters).

The Finding: It didn't matter how big the brain was. All of them failed.
The Metaphor: It's like giving a giant library to a person who is blind. No matter how many books (data) they have, if they can't see the shape of the object in front of them, they can't identify it. The problem isn't the size of the library; it's that the "eyes" (the visual part of the AI) are broken when it comes to pure geometry.

4. The "Hallucination" Habit

When the AI gets confused because it can't see the texture, it doesn't say, "I don't know." Instead, it starts guessing based on what it remembers from the internet.

If the AI sees a blurry outline that might be a dog, it guesses "Afghan Hound" because it has seen that phrase millions of times in its training data.
It's like a student taking a test who doesn't know the answer, so they just write down the most popular word they've heard in class, hoping it's right.

Why Does This Matter?

The paper argues that current AI is brittle.

Real World: In the real world, things don't always look perfect. Lighting changes, shadows appear, and objects get covered in mud or snow. If an AI relies on "stripes" to find a tiger, it might miss a tiger in the snow.
The Goal: We want AI that understands geometry—the actual structure of the world—so it can work in any condition, not just when the lighting is perfect and the textures are clear.

The Takeaway

The "BareBones" benchmark is a wake-up call. It shows that while our AI is amazing at reading and describing pictures, it doesn't truly understand the shapes of the objects in them. It's like a person who can describe a car in great detail but can't recognize it if you paint it black and white.

To build truly intelligent machines, we need to teach them to see the skeleton of the world, not just the skin.

1. Problem Statement

Despite the remarkable zero-shot capabilities of modern Vision-Language Models (VLMs) in tasks like scene understanding and reasoning, it remains unclear whether these models genuinely comprehend geometric structure or merely rely on RGB textures, color histograms, and contextual priors as statistical shortcuts.

The Gap: Existing benchmarks (e.g., COCO, ImageNet) conflate semantic reasoning with texture mapping. A model can identify a tiger based on its stripes without understanding its shape.
The Limitation: Current evaluations fail to isolate geometric reasoning because loose bounding boxes leak environmental cues, and pixel-level annotations are often imprecise. This masks fundamental visual deficits, leading to brittle models that fail under novel appearances or when surface patterns are removed.

2. Methodology: The BareBones Benchmark

The authors introduce BareBones, a zero-shot benchmark designed to stress-test pure geometric shape comprehension by completely removing RGB information.

A. Dataset Curation

The benchmark utilizes six high-fidelity segmentation taxonomies, converting them into binary silhouettes (pure structural outlines):

ImageNet-S: 100 base classes with distinct outlines.
DIS5K & ThinObject5K: Complex boundaries, fine appendages, and thin-wire geometries.
PASCAL VOC: Articulated poses and environmental occlusions.
CUB-200: 200 bird species where silhouettes differ only in subtle anatomical details (beak curvature, tail length).
WTP-Bench (Flagship): A novel dataset of 1,160 Pokémon silhouettes spanning 8 generations. This serves as an extreme fine-grained test where species share base geometries but differ in intricate localized features (horns, wings, tails).

B. Canonicalization & Evaluation Pipeline

Input Processing: For every source image, the dominant segmentation mask is binarized to a pure structural silhouette (white foreground, black background). All color, texture, shading, and background context are eliminated.
Standardization: Images are cropped to bounding boxes and resized to a maximum dimension of 200px to standardize input across encoders.
Evaluation Modes:
1. High-Quality (HQ): Full-color images (establishes the texture upper bound).
2. Silhouette (Shape): Binarized masks only (tests pure geometric reasoning).
Protocol: Strict zero-shot setting ( $T=0.0$ for APIs, 100-token output cap). Accuracy is measured via top-1 exact string match.

C. Model Scope

The study evaluates 26 state-of-the-art VLMs, including:

7 Proprietary Models: e.g., GPT-4.1, Gemini 2.5 Pro, Claude Sonnet 4.5.
19 Open-Weight Models: Ranging from 1B to 26B parameters (e.g., LLaVA, Qwen-VL, InternVL, SmolVLM).

3. Key Findings: The "Texture Bias Cliff"

The evaluation reveals a consistent, severe performance collapse when RGB textures are removed, a phenomenon the authors term the Texture Bias Cliff.

A. Universal Performance Collapse

Across Architectures: All models, regardless of size (1B–26B) or type (proprietary vs. open-weight), suffer a drastic drop in accuracy when moving from HQ to Silhouette inputs.
Quantitative Evidence:
- On WTP-Bench, the best proprietary model (GPT-4.1) drops from 78.97% (HQ) to 49.48% (Silhouette).
- Open-weight models perform catastrophically: Qwen3-VL 4B drops from 13.36% to 3.10%, and most architectures fall below 2.2% on silhouettes.
- On CUB-200, 137 out of 200 bird species achieve 0% accuracy across all models in silhouette mode.

B. Scale Does Not Solve the Problem

Parameter Scaling Failure: Increasing model size from 1B to 26B does not mitigate the cliff. Models cluster between 1–5% silhouette accuracy regardless of parameter count, indicating the failure is architectural (rooted in the visual encoder) rather than a lack of scale.

C. Error Analysis: Hallucination of Priors

When vision encoders fail to parse silhouettes, language decoders default to pre-training corpus statistics rather than structural deduction.

Generation Bias: On WTP-Bench, 79.2% of open-weight misclassifications default to Generation 1 Pokémon, despite Gen 1 comprising only 19.6% of the dataset. This mirrors human familiarity gradients but confirms models are hallucinating based on text frequency, not visual analysis.
Specific Hallucinations: Models frequently hallucinate high-frequency classes like "Afghan Hound" or "American Crow" regardless of the actual silhouette shape.

D. Morphological Vulnerabilities

Complex Geometries: Ghost-, Bug-, and Dragon-type Pokémon (amorphous, non-contiguous, or thin-wire geometries) are the hardest to identify.
Form Variants: Mega-evolved and Gigantamax forms incur the steepest drops, suggesting models cannot handle geometric transformations or relational structures (e.g., recognizing a Mega form as a modification of a base form).

4. Key Contributions

BareBones Benchmark: A rigorous, zero-shot benchmark that isolates geometric reasoning by eliminating all texture and context cues across six diverse taxonomies.
WTP-Bench Dataset: A novel, fine-grained visual puzzle (Pokémon silhouettes) that establishes a human-tractable ceiling (~70% for fans) while exposing a near-zero capability gap in current frontier models.
The Texture Bias Cliff: The identification and quantification of a universal, architecture-agnostic failure mode where VLMs lose geometric grounding entirely when deprived of RGB textures.
Architectural Diagnosis: Evidence that current VLM failures are not due to insufficient parameter scale but are fundamental flaws in how Vision Transformers (ViTs) encode shape versus texture.

5. Significance and Implications

Redefining Evaluation: The paper argues that current benchmarks are "too easy" because they allow models to cheat using texture. BareBones provides a necessary yardstick for genuine geometric grounding.
Real-World Robustness: The findings suggest that current VLMs are brittle and may fail in real-world scenarios involving novel lighting, occlusion, or non-canonical appearances where texture cues are unreliable.
Future Directions: The results call for a shift in training objectives to prioritize shape bias over texture bias. The authors warn against training directly on BareBones data (which would reduce a structural intelligence test to trivial 2D pattern matching) and suggest exploring intermediate conditions (e.g., edge maps) to disentangle resolution effects from texture dependence.

In conclusion, BareBones demonstrates that despite impressive multimodal capabilities, current VLMs lack a fundamental understanding of 3D geometry and structural relationships, relying instead on statistical shortcuts derived from surface textures.