💬 NLP

Disparities In Negation Understanding Across Languages In Vision-Language Models

This paper introduces the first human-verified multilingual benchmark for negation understanding in vision-language models, revealing that while correction methods like SpaceVLM improve performance, their effectiveness varies significantly across typologically diverse languages, highlighting critical disparities in how linguistic properties interact with model fairness.

Original authors: Charikleia Moraitaki, Sarah Pan, Skyler Pulling, Gwendolyn Flusche, Kumail Alhamoud, Marzyeh Ghassemi

Published 2026-04-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Charikleia Moraitaki, Sarah Pan, Skyler Pulling, Gwendolyn Flusche, Kumail Alhamoud, Marzyeh Ghassemi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart robot assistant that looks at pictures and tries to describe what it sees. You might ask it, "Is there a boat in this picture?" If the answer is "No," the robot needs to understand that the word "no" completely flips the meaning.

This paper is about a big problem: These robots are terrible at understanding "No," and they are even worse at it when you speak a language other than English.

Here is the breakdown of the story, using some simple analogies:

1. The "Yes-Man" Robot (Affirmation Bias)

The researchers discovered that these Vision-Language Models (VLMs) have a habit of being "Yes-Men."

The Analogy: Imagine a robot that is so eager to please that if you show it a picture of a beach and say, "There is no boat here," the robot ignores the word "no." It sees the word "boat" and the picture of the beach, and it confidently says, "Yes, there is a boat!"
The Danger: This isn't just a funny mistake. In real life, like in a hospital, if a doctor asks, "Is there no cancer?" and the robot says "Yes, there is cancer" (because it missed the "no"), the consequences could be life-threatening.

2. The "English-Only" Blind Spot

Until now, everyone tested these robots only in English. It's like testing a car only on smooth, dry highways in California and assuming it will drive perfectly on icy, muddy roads in Russia or sandy dunes in the Middle East.

The Problem: English is actually quite simple when it comes to saying "no." You just add a little word like "not" or "no" in front of the sentence.
The Reality: Other languages are much more complex.
- Greek might change the whole verb structure to say "it does not exist."
- Arabic writes right-to-left and attaches the "no" to the word like a sticker.
- Chinese uses tiny particles that act like traffic signs, changing the meaning of the whole sentence.
The Finding: The researchers built a test (a "benchmark") with seven different languages (including English, Chinese, Arabic, Greek, Russian, Spanish, and Tagalog). They found that while the robots were okay at saying "no" in English, they were often completely lost in other languages. In some cases, they performed worse than if a human had just guessed randomly.

3. The "Universal Fix" That Isn't Universal

The researchers tried a new trick called SpaceVLM. Think of this as a "translator" or a "spell-checker" designed to help the robot understand negative sentences better.

The Experiment: They applied this fix to all seven languages.
The Result: It worked like magic for languages that are similar to English (like Spanish, Greek, and Tagalog). The robot suddenly got much smarter.
The Twist: For languages with complex "no" structures (like Russian, Arabic, and Chinese), the fix didn't work well. In fact, for some, it made things worse.
The Metaphor: Imagine you have a tool designed to tighten a standard screw. It works perfectly on standard screws (English-like languages). But if you try to use that same tool on a square bolt or a hex nut (complex languages), it doesn't just fail; it might strip the bolt entirely. The "fix" was built for English logic, so it breaks when applied to different linguistic logic.

4. Why This Matters (The "Digital Divide")

The paper concludes that we cannot just build AI for English speakers and assume it works for everyone else.

The Takeaway: If we deploy these robots globally (in hospitals, self-driving cars, or security systems), we are currently giving high-quality service to English speakers and low-quality, unreliable service to everyone else.
The Solution: To make AI fair, we need to stop treating all languages as if they are just "English with a different accent." We need to understand the unique "grammar DNA" of every language and build specific tools for them, rather than trying to force one English-based solution onto the whole world.

In short: These robots are great at seeing pictures, but they are terrible at understanding the word "no," especially if you don't speak English. Fixing this requires us to stop using "one-size-fits-all" solutions and start respecting the unique way every language says "no."

1. Problem Statement

Vision-Language Models (VLMs) suffer from a systematic affirmation bias, where they tend to match positive captions (e.g., "a boat is present") to images even when the correct description contains negation (e.g., "no boat"). While this failure mode has been documented in English, existing evaluations are monolingual.

The core problem addressed is the lack of understanding regarding how negation bias manifests across different linguistic typologies. Negation is expressed differently across languages through:

Morphology: Some languages use simple adverbial particles (e.g., English "not"), while others use complex verbal negation or cliticized markers (e.g., Arabic, Greek).
Script and Direction: Variations include Latin vs. non-Latin scripts and Left-to-Right vs. Right-to-Left writing systems.
Structure: Differences in word order and cliticization patterns.

The authors argue that solutions developed for English may not translate equitably to other linguistic communities, potentially exacerbating the "digital language divide" in safety-critical domains like medical imaging (e.g., distinguishing "no pleural effusion" from "pleural effusion present").

2. Methodology

Benchmark Construction

The authors created the first human-verified multilingual negation benchmark, extending the English NegBench dataset (5,914 image-caption sets from COCO).

Languages: Seven typologically diverse languages were selected:
- Adverbial/Particle-based: English, Spanish, Greek, Tagalog.
- Morphologically Complex: Russian, Arabic.
- Isolating with distinct particles: Mandarin Chinese.
Translation Process: Translations were generated via Google Translate and rigorously verified by native speakers (30 samples per language) to ensure semantic faithfulness, correct negation markers, and natural phrasing.

Models Evaluated

Three contrastive VLMs representing different training regimes were tested:

CLIP: Primarily English-trained.
SigLIP: Multilingual training.
MultiCLIP: Multilingual training.
SpaceVLM: A proposed negation correction method (Ranjbar et al., 2025) was applied to all models. SpaceVLM computes hybrid embeddings by decomposing captions into affirmative and negated components using a threshold ( $\tau=0.92$ ) derived from English data.

Evaluation Metric

Performance was measured using Top-1 Accuracy on a 4-way caption ranking task (chance level = 25%).

3. Key Contributions

First Multilingual Negation Benchmark: Introduction of a human-verified dataset spanning seven typologically diverse languages, enabling the measurement of cross-lingual negation gaps.
Empirical Evidence of Inequity: Demonstration that even explicitly multilingual models exhibit significant performance disparities based on language structure.
Typological Analysis of Corrections: Discovery that the effectiveness of negation correction methods (like SpaceVLM) is not uniform but correlates strongly with linguistic typology (specifically, how negation is morphologically expressed).

4. Key Results

Baseline Performance Disparities

CLIP: Exhibited extreme disparity. While it achieved 39.3% accuracy on English, it performed below chance on Arabic (15.7%), Tagalog (11.8%), and Greek (18.0%). The gap between best and worst performance was 27.5 percentage points.
SigLIP & MultiCLIP: Multilingual training reduced the variance (SigLIP gap: 7.4pp; MultiCLIP gap: 4.4pp) but did not eliminate the problem. Even the best model (MultiCLIP) only achieved a mean accuracy of 41.2%, indicating that multilingual pretraining alone is insufficient for negation understanding.

Impact of SpaceVLM (Correction Method)

Applying SpaceVLM revealed a distinct typological pattern:

High Gain: Languages using independent adverbial particles (English, Spanish, Greek, Tagalog) saw substantial improvements (+9.0 to +38.9 percentage points). For example, Tagalog on CLIP jumped from 11.8% to 50.7%.
Variable/Low Gain: Languages with morphologically complex negation (Russian, Arabic) or distinct semantic particles (Chinese) showed minimal gains or even performance degradation.
- Example: SpaceVLM reduced SigLIP's accuracy on Chinese by 19.5 points and on Russian by 10.6 points.
Robustness: This pattern persisted even when applied to models fine-tuned specifically for English negation (NegCLIP, ConCLIP), suggesting the issue is fundamental to how these languages encode negation relative to English-centric architectures.

5. Significance and Implications

Fairness in AI Deployment: The study highlights that "one-size-fits-all" solutions for VLM alignment can inadvertently disadvantage speakers of languages with non-English negation structures.
Linguistic Structure Matters: The effectiveness of technical fixes (like SpaceVLM) is contingent on linguistic typology. English-based parsing strategies fail to generalize to languages with different morphological negation systems.
Future Directions:
- Data: Need for negation-rich, multilingual pretraining data.
- Architecture: Development of tokenization strategies that preserve negation scope across morphological systems.
- Calibration: Correction methods (e.g., SpaceVLM thresholds) should be tuned per typological group rather than globally.
- Auditing: Fairness audits must move beyond simple accuracy metrics to include linguistic structural analysis.

In conclusion, the paper establishes that achieving equitable VLM performance requires moving beyond English-centric benchmarks and acknowledging that linguistic typology fundamentally shapes model behavior and the efficacy of alignment interventions.