💻 computer science

When Negation Is a Geometry Problem in Vision-Language Models

This paper proposes a Multimodal LLM-based evaluation framework to rigorously assess negation understanding in Vision-Language Models and demonstrates that negation-aware behavior can be achieved without fine-tuning by identifying and manipulating a specific negation direction in the CLIP embedding space via test-time representation engineering.

Original authors: Fawaz Sammani, Tzoulio Chamiti, Paul Gavrikov, Nikos Deligiannis

Published 2026-03-24

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Fawaz Sammani, Tzoulio Chamiti, Paul Gavrikov, Nikos Deligiannis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart librarian named CLIP. This librarian has read millions of books and looked at millions of photos. If you ask, "Show me a picture of a dog on the grass," CLIP is amazing at finding it.

But if you ask, "Show me a picture of a dog NOT on the grass," CLIP gets confused. It often ignores the word "NOT" and just shows you a dog on the grass anyway. It's like asking a chef, "Make me a sandwich without cheese," and they hand you a cheese sandwich because they only heard the word "sandwich."

This paper is about fixing that confusion, but with a twist: the authors realized the old ways of testing if CLIP was getting better were broken, and they found a clever shortcut to fix the problem without retraining the librarian.

Here is the story of their discovery, broken down into simple parts:

1. The Broken Test (The "Fake Negative" Problem)

For a long time, researchers tried to teach CLIP to understand "NOT" by showing it thousands of examples. They would generate fake sentences like "a car with no wheels" and show it pictures.

But the authors found a huge flaw in how they tested if this worked.

The Analogy: Imagine you are testing a security guard. You ask, "Is there a red car in the parking lot?" The guard points to a red car. You give them a gold star.
The Problem: Now, you ask, "Is there a red car with NO tires?" The guard points to a red car with no tires. But wait! The test database only marked one specific car as the "correct" answer. Even though the guard found a perfectly valid car with no tires, the test said, "Wrong! That wasn't the exact car we wrote down."
The Result: The researchers realized that standard tests were full of "false negatives." They were punishing the models for finding good answers that just weren't the exact answer the computer expected. It was like grading a student wrong for solving a math problem correctly but using a different method than the textbook.

2. The New Judge (The "Smart AI" Referee)

To fix the broken test, the authors introduced a new referee: a Multimodal Large Language Model (MLLM). Think of this as a super-smart, human-like AI that can look at a picture and answer simple questions.

Instead of just checking if the image matches a database label, they ask the Smart AI two questions:

"Does this picture match the general idea?" (e.g., Is there a dog?)
"Does it follow the 'NOT' rule?" (e.g., Is the dog not on the grass?)

If the Smart AI says "Yes" to both, the model gets a point. This is a much fairer way to see if the model actually understands the concept of "NOT."

3. The Hidden Compass (The "Negation Vector")

Once they had a fair test, they asked a big question: Does CLIP already know how to say "NOT," or do we have to teach it from scratch?

They discovered that CLIP does already know.

The Analogy: Imagine CLIP's brain is a giant 3D map. Every sentence is a dot on this map. The authors found that there is a specific direction (like a compass needle) in this map that points toward "Negation."
When you say "Dog," the dot is in one spot. When you say "No Dog," the dot is in a slightly different spot, but it's always in the same direction relative to the original.
The problem is that CLIP isn't naturally using this compass needle when it's trying to find pictures. It's ignoring the "No" part.

4. The Magic Nudge (Steering, Not Retraining)

Most previous research tried to fix CLIP by feeding it millions of new examples (retraining), which is expensive and slow. It's like trying to teach a grown adult a new language by making them read every book in the library again.

The authors found a much easier way: Representation Steering.

The Analogy: Imagine you are driving a car (CLIP) and you want to go North (Negation). Instead of rebuilding the engine or taking a new driving school course, you just gently nudge the steering wheel in the direction of North.
They found that "Negation Direction" vector. When a user asks a question with "NOT," they simply add a tiny mathematical push to the computer's internal thought process, nudging it toward that "Negation" direction.
The Result: Suddenly, CLIP starts understanding "NOT" perfectly, without needing to be retrained or fed new data. It's like giving the librarian a pair of glasses that suddenly makes the word "NOT" pop out in bright red letters.

5. The Surprise: Why "Training" Actually Hurt

The most surprising part of the paper is what they found when they compared their "Magic Nudge" method against the models that were retrained.

The Retrained Models: These models were trained on millions of synthetic examples. They got good at the specific test questions but failed when asked about weird, new things (like "a book made of glass"). They had memorized the patterns but lost their flexibility.
The "Magic Nudge" Model: Because they didn't retrain the model, it kept its original ability to understand the world. When they just nudged the steering wheel, it handled both normal questions and weird, new questions much better.

The Big Takeaway

The paper teaches us two main lessons:

Don't trust the old tests: Just because a model gets a high score on a standard test doesn't mean it actually understands logic. You need a smarter referee (like the MLLM) to check if it really gets the "NOT" part.
Sometimes, less is more: You don't always need to retrain a massive AI with millions of examples to fix a specific problem. Sometimes, the AI already has the knowledge hidden inside; you just need to find the right "knob" to turn and nudge it in the right direction.

In short: The authors didn't build a new engine; they just found the right way to steer the one they already had.

1. Problem Statement

Vision-Language Models (VLMs) like CLIP struggle significantly with understanding negation in text queries (e.g., distinguishing "a dog on the grass" from "a dog not on the grass").

Current Limitations: Existing solutions rely on data-centric approaches, where synthetic negation datasets are generated to fine-tune CLIP.
Evaluation Flaws: These approaches are typically evaluated using standard retrieval metrics (e.g., Recall@K). The authors identify two critical flaws in this evaluation paradigm:
1. False Negatives: Synthetic data pipelines often label only one image as ground truth, ignoring other valid images that satisfy the negated query. This penalizes models that retrieve semantically correct but unlabelled images.
2. Model Collapse: Fine-tuning on synthetic negation data can cause models to "collapse," retrieving images entirely unrelated to the query or collapsing all negated queries into a single embedding point, rendering them indistinguishable.

2. Methodology

The paper proposes a two-pronged approach: a new evaluation framework and a novel inference-time intervention technique.

A. Evaluation Framework: MLLM-as-a-Judge

To overcome the limitations of retrieval-based metrics, the authors propose using Multimodal Large Language Models (MLLMs) as judges.

Protocol: For a retrieved image and a negated query, the MLLM answers two sequential questions:
1. Semantic Check ( $q_r$ ): "Is the image contextually and semantically correct?" (e.g., Is there a dog?)
2. Negation Check ( $q_n$ ): "Is the negated object absent?" (e.g., Is the dog not running?)
Advantage: This unsupervised method evaluates the logic of the retrieval rather than relying on potentially flawed ground-truth annotations, allowing evaluation on any database size.

B. Core Investigation: The "Negation Direction"

The authors hypothesize that negation information is already encoded in CLIP's embedding space but is not utilized during standard inference.

Discovery: Using a linear binary classifier on hidden states from the CLIP text encoder (specifically the <eos> token), they demonstrated that original and negated captions are linearly separable with >99% accuracy in intermediate layers.
Intervention (Representation Steering): Instead of fine-tuning, they manipulate the embeddings at inference time:
1. Train a lightweight linear classifier on a small set of negated text pairs (4,000 samples) to find a negation direction vector ( $W_{dir}$ ).
2. Steer the text embedding ( $h_l$ ) toward this direction using the formula:
  $h_l' = (1 - \alpha)h_l + \alpha W_{dir}||h_l||$
  Where $\alpha$ is a hyperparameter controlling the steering strength.
3. This geometric manipulation forces the model to attend to negation cues without modifying model weights or requiring large-scale retraining.

3. Key Contributions

Critique of Current Metrics: Identified that standard retrieval metrics fail to distinguish between semantic matching and genuine negation understanding, often masking model failures due to false negatives in synthetic benchmarks.
MLLM-as-a-Judge: Introduced a robust, unsupervised evaluation protocol that separates semantic correctness from negation logic.
Geometric Steering: Proved the existence of a latent "negation direction" in CLIP embeddings and demonstrated that test-time representation steering can effectively enable negation understanding without fine-tuning.
Generalization: Showed that steering outperforms fine-tuned baselines on distribution-shifted data (non-common object compositions), whereas fine-tuned models often overfit and fail.

4. Experimental Results

The authors evaluated their method against baselines (Original CLIP, ConCLIP, NegCLIP, CLIP-CC12M) on three benchmarks:

NegBench (Standard Synthetic Benchmark):
- Using MLLM-as-a-judge, the Steering method achieved the highest performance (Top-1: 54.3% with Qwen3-VL-32B), outperforming NegCLIP (46.0%) and CLIP-CC12M (53.1%).
- Standard retrieval metrics (Recall@K) showed all models improving, but failed to reveal that fine-tuned models often collapsed or retrieved irrelevant images, a failure mode exposed only by the MLLM judge.
SimpleNeg (Controlled Unit Test):
- The steering approach achieved near-perfect performance (Top-1: 54.3% - 55.0% across judges), significantly surpassing all fine-tuned baselines.
- This confirmed that the negation direction is explicitly identifiable and manipulable.
N-COCO (Non-Common Objects / Distribution Shift):
- Crucial Finding: Fine-tuned models (NegCLIP, CLIP-CC12M) suffered performance drops (R@1 dropped to 0.50) when tested on uncommon object compositions (e.g., "a book not made of paper").
- Steering Performance: The steering method maintained robustness, achieving R@1 of 0.80.
- Conclusion: Fine-tuning on synthetic data leads to overfitting on specific patterns, whereas geometric steering preserves the model's generalization capabilities while adding negation logic.

5. Significance and Conclusion

This paper fundamentally shifts the perspective on negation in VLMs from a data problem to a geometry problem.

Efficiency: The proposed solution requires only 4,000 text samples (0.03% of data used in previous approaches) and zero fine-tuning of the massive CLIP model.
Robustness: It demonstrates that representation steering is more robust to distribution shifts than data-centric fine-tuning, which risks distorting the embedding space.
Implication: The findings suggest that large pre-trained models already possess the necessary internal representations for complex logical operations like negation; the challenge lies in how to access and steer these representations effectively at inference time.

The work advocates for a paradigm shift where lightweight, representation-level interventions replace heavy retraining for specific logical capabilities in multimodal models.