💬 NLP

More Than Meets the Eye: Measuring the Semiotic Gap in Vision-Language Models via Semantic Anchorage

This paper introduces the DIVA benchmark and the Semantic Alignment Gap metric to demonstrate that Vision-Language Models exhibit a consistent literal superiority bias, where increased visual fidelity and model scale fail to resolve struggles with abstract idiomatic meanings, suggesting that iconographic abstraction is necessary for improved compositional understanding.

Original authors: Wei He

Published 2026-04-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Wei He

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: AI is Too Literal

Imagine you show a picture of a "Red Carpet" to a human. They instantly understand it's a metaphor for fame, luxury, or a special event. They don't just see a piece of red fabric on the floor.

Now, show that same picture to a current AI (a Vision-Language Model). The AI gets stuck on the details. It sees the texture of the carpet, the lighting, and the color red. It struggles to "get the joke" or the metaphor. It's like a student who is so good at memorizing the dictionary definitions of words that they can't understand a poem.

The researchers call this the "Literal Superiority Bias." The AI prefers the boring, physical truth over the clever, abstract meaning.

The Solution: The "DIVA" Benchmark

To test if this is true, the researchers created a new test called DIVA (Distilled Idiomatic Visual Abstraction).

Think of it like this:

The Old Way (High Fidelity): Showing the AI a hyper-realistic, 4K photo of a "Red Carpet." It's so detailed (shadows, wrinkles, dust) that the AI gets distracted by the "noise" and forgets the meaning.
The New Way (DIVA): Showing the AI a stick-figure drawing or a simple icon of a red carpet. It's a "schematic" image—clean, simple, and stripped of all the distracting details.

The Analogy:
Imagine trying to explain a complex idea to a friend.

High Fidelity is like giving them a 100-page novel with footnotes, illustrations, and a detailed map. They get lost in the details.
DIVA (Iconography) is like drawing a simple stick figure on a napkin. It's crude, but it cuts out the noise and hits the core idea immediately.

The Experiment: What Happened?

The researchers tested 8 different AI models (some open-source, some from big companies like OpenAI and Anthropic) using both the realistic photos and the simple drawings.

They measured something called the "Semantic Alignment Gap."

High Gap: The AI is confused. It thinks the literal meaning is totally different from the idiomatic meaning.
Low Gap: The AI understands that the picture represents the idea, not just the object.

The Results:

Size Doesn't Matter: Making the AI bigger (giving it more "brain power") didn't fix the problem. Even the giant, super-smart models still got stuck on the literal meaning when looking at realistic photos.
Simplicity Wins: When the researchers switched the photos to simple drawings (DIVA), the AI's performance skyrocketed. The "gap" disappeared. The models suddenly understood the metaphors.

The Metaphor:
It's like trying to hear a whisper in a rock concert.

Realistic Photos are the rock concert. The AI is trying to hear the meaning (the whisper), but the visual details (the loud music) are drowning it out.
Simple Drawings are turning the music off. Suddenly, the whisper is clear. The AI didn't get smarter; the environment just stopped confusing it.

The "Why": Cognitive Interference

The paper suggests that current AI models are trained to be "photorealistic." They are obsessed with texture, lighting, and shadows. When they see a "Web Site," they see a spider web. When they see "Eye Candy," they see actual eyes and candy.

The researchers argue that hyper-realism is actually a trap. It tricks the AI into thinking about physics instead of language. By removing the "realism" and using symbols (like a cartoon), we force the AI to stop looking at the surface and start looking at the meaning.

The Takeaway

This paper tells us that to make AI truly understand human language and metaphors, we might need to stop feeding it perfect, realistic photos. Instead, we should teach it to read symbols and diagrams, just like we do when we read a map or a comic book.

In short: If you want an AI to understand a joke, don't show it a high-definition photo. Show it a doodle. Sometimes, less detail means more understanding.

1. Problem Statement

Vision-Language Models (VLMs) have achieved remarkable proficiency in generating and interpreting photorealistic imagery. However, they exhibit a fundamental cognitive limitation known as "Literal Superiority Bias." When processing abstract concepts, particularly Noun Compounds (NCs) with idiomatic meanings (e.g., "Eye Candy" meaning attractive people, not literal candy), models struggle to move beyond high-fidelity visual details (iconicity) to grasp abstract symbolic meanings.

The core issue is a Semiotic Gap:

Iconicity: Images are typically interpreted as depictions of physical reality (relying on texture, lighting, and clutter).
Symbolism: Language relies on conventional codes where meaning is abstract.
The Conflict: Current VLMs prioritize high-fidelity visual cues over semantic alignment. When presented with an idiom, the model often defaults to a literal interpretation based on the constituent objects (e.g., seeing an eye and candy) rather than the intended metaphorical meaning, even when the text anchor is ambiguous.

2. Methodology

The authors propose a framework to bridge this gap by shifting the visual input from "simulation" to "symbolism" and introducing a unified metric to measure the resulting alignment.

A. The DIVA Benchmark (Distilled Idiomatic Visual Abstraction)

To control for visual realism, the authors created DIVA, a benchmark that replaces high-fidelity photorealistic images with schematic, low-detail iconographic renderings.

Process: A two-stage pipeline using automated generation (via gemini-3-pro-image-preview) followed by human verification.
Transformation: High-fidelity images are transformed into "Iconographic Symbolism" by stripping away textures, shadows, and background clutter while preserving the semantic core.
Data Structure: The benchmark contains 200 English Noun Compounds. For each, it provides paired visualizations:
- Literal Rendering ( $v_{lit}$ ): A schematic depiction of the constituent nouns.
- Idiomatic Rendering ( $v_{id}$ ): A schematic depiction of the conventional metaphorical meaning.
Goal: To test if reducing visual "noise" (semiotic superfluity) forces the model to rely on symbolic reasoning rather than physical simulation.

B. The Semantic Alignment Gap ( $\Delta$ ) and Signed Bias ( $b(t)$ )

The authors introduce an architecture-agnostic metric to quantify the divergence between literal and idiomatic grounding.

Signed Literal Bias ( $b(t)$ ): Measures the direction of preference.
$b(t) = S(v_{lit}, t) - S(v_{id}, t)$
Where $S$ is a scoring function. A positive value indicates a preference for the literal meaning; a negative value indicates idiomatic preference.
Semantic Alignment Gap ( $\Delta$ ): Measures the magnitude of the divergence (strength of the bias).
$\Delta(t) = |b(t)|$
A lower $\Delta$ indicates the model treats both interpretations more equally (or correctly identifies the intended sense when context is provided), suggesting reduced literal drift.

C. Tri-Fold Scoring Implementation

To ensure comparability across different model types, the scoring function $S$ is instantiated in three ways:

Discriminative Models (Open-Weights): Uses Intrinsic Alignment (Cosine similarity between text and image embeddings in latent space, e.g., CLIP, SigLIP).
Open-Source Generative Models: Uses White-Box Confidence (Likelihood of Idiomatic Distinction - LID). This calculates the probability of a "Yes/No" response to whether an image matches the idiomatic sense using token logits.
Proprietary/Closed-Source Models: Uses Extrinsic Confidence (Self-reported confidence scores and behavioral choice frequency via forced-choice prompting).

3. Key Contributions

DIVA Dataset: A controlled benchmark of 1,000 iconographic images designed to isolate the effect of visual fidelity on semantic understanding.
Unified Metric ( $\Delta$ ): An architecture-agnostic framework to measure the "Literal Bias" across discriminative, open-generative, and closed-generative models.
Theoretical Framework: The concept of Iconographic Abstraction via Semantic Anchorage, positing that reducing visual fidelity (removing "semiotic noise") enhances symbolic clarity and reduces cognitive interference.

4. Results and Analysis

The authors evaluated 8 state-of-the-art VLMs (including SigLIP, EVA-CLIP, InternVL3, Qwen2.5-VL, GPT-5, and Claude 4.5) under two conditions: High-Fidelity (Photo) and Iconographic (DIVA).

Consistent Literal Bias: All models exhibited a positive $b(t)$ (literal preference) in high-fidelity conditions. Model scale alone (e.g., moving from 7B to 78B parameters) did not resolve this bias.
Reduction of the Gap: Shifting from photorealistic to iconographic inputs consistently reduced $\Delta$ $Δ$ across all architectural families.
- Example: GPT-5's alignment gap dropped from $\Delta \approx 0.065$ (Photo) to $\approx 0.021$ (Icon).
- Example: Discriminative models (SigLIP) saw a reduction from $\Delta \approx 0.245$ to $\approx 0.178$ .
Improved Accuracy: In a sense-specified 5-way selection task (where the intended meaning was explicitly described), accuracy improved significantly with iconographic inputs:
- Discriminative: 42.3% $\to$ 58.7%
- Open-Generative: 61.8% $\to$ 74.2%
- Proprietary: 78.5% $\to$ 91.3%
Cognitive Interference: The results suggest that high-fidelity textures act as a confounding variable. Models trained on natural images over-optimize for physical reconstruction, which suppresses their ability to process abstract metaphors.

5. Significance and Implications

Rethinking Visual Input: The paper challenges the assumption that higher visual fidelity always leads to better understanding. It suggests that for abstract reasoning, visual minimalism may be superior to photorealism.
Accessibility and Communication: The findings align with Augmentative and Alternative Communication (AAC) needs, where simplified symbols are often more effective for conveying meaning than complex images.
Future Directions: The authors propose that future VLMs should be trained to "read" schematic glyphs before attempting to "render" high-fidelity realities. They suggest a Contrastive Idiom Tuning (CIT) framework to explicitly teach models to distinguish between literal and symbolic imagery.
Limitations: The study is currently limited to English Noun Compounds and specific artistic styles (flat design). The authors acknowledge that tokenization strategies and cultural metaphors may yield different results in other languages.

In conclusion, the paper demonstrates that the "Literal Superiority Bias" in VLMs is not just a lack of reasoning capability but a result of cognitive interference from hyper-realistic imagery. By anchoring interpretation in schematic abstraction, models can achieve significantly better alignment with abstract, idiomatic meaning.