🤖 machine learning

Is CLIP ideal? No. Can we fix it? Yes!

This paper proves that CLIP's latent space fundamentally cannot simultaneously handle basic content, attribute binding, spatial relationships, and negation, and proposes Dense Cosine Similarity Maps (DCSMs) as a principled alternative that overcomes these geometric limitations by preserving the semantic topology of image patches and text tokens to achieve superior performance.

Original authors: Raphi Kang, Yue Song, Georgia Gkioxari, Pietro Perona

Published 2026-03-12

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Raphi Kang, Yue Song, Georgia Gkioxari, Pietro Perona

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Smart" Camera That Gets Confused

Imagine you have a super-smart camera (called CLIP) that has read every book and looked at every photo on the internet. It's amazing at matching pictures to words. If you show it a picture of a cat and say "cat," it gives a high score. If you say "dog," it gives a low score.

But, this camera has a weird blind spot. It's great at recognizing what is in the picture, but it gets very confused when you ask it about how things are arranged or what is missing.

The Problem:
If you show the camera a picture of a red circle and a blue triangle, and you ask it, "Is this a red triangle and a blue circle?" the camera often says, "Yes! That looks very similar!" It can't tell the difference between the two because it treats the words like a grocery list rather than a sentence. It sees "red," "triangle," "blue," and "circle," and just checks if they are all present, ignoring the order and the relationships.

It also fails at:

Negation: If you say "No red circle," it might still think you want a red circle because it sees the word "red" and "circle."
Spatial Logic: If you say "The dog is under the table," it struggles to understand that the dog is below, not above.

The "Why": The Geometry of a Round Room

The authors of this paper asked a deep question: Is this confusion just a bug we can fix with more training, or is it a fundamental flaw in how the camera thinks?

They proved mathematically that the camera's "brain" is built like a round room (a hypersphere). In this room, every concept is a point on the wall.

To make the camera understand "Red Circle," you have to place a point on the wall.
To make it understand "Red Triangle," you place another point.
To make it understand "Red Circle AND Blue Triangle," the camera tries to find a spot that is halfway between "Red Circle" and "Blue Triangle."

The Analogy: Imagine you are trying to mix paints.

If you mix Red and Blue, you get Purple.
But in this camera's brain, mixing "Red Circle" and "Blue Triangle" doesn't create a new, unique concept. It just creates a muddy average that looks like "Red Triangle" and "Blue Circle" at the same time.

The authors proved that it is mathematically impossible to arrange these points on the round wall so that the camera gets the order, the location, and the "not" logic right at the same time. The shape of the room itself is the problem. You can't fix the room just by painting it better; the room is the wrong shape for the job.

The Solution: The "Mosaic" Instead of the "Summary"

Since we can't change the shape of the room, the authors asked: Can we look at the room differently?

Instead of asking the camera for a single "summary score" (like a final grade), they proposed looking at the entire mosaic of the image and the text.

The Old Way (CLIP):
The camera looks at the whole image and the whole sentence, squishes them into two single dots, and measures the distance between them.

Result: "Red Circle + Blue Triangle" and "Red Triangle + Blue Circle" end up as the same dot.

The New Way (DCSM - Dense Cosine Similarity Maps):
The authors say, "Don't squish it! Keep the details."

They take every single word in the sentence (like "red," "circle," "above").
They take every single tiny patch of the image (like the top-left corner, the bottom-right corner).
They create a giant grid (a map) showing how every word matches every patch.

The Analogy:

Old Way: It's like a teacher reading a student's essay and giving it a single grade of "B." The teacher doesn't know why it got a B.
New Way (DCSM): It's like the teacher highlighting every specific sentence in the essay and drawing lines to the specific facts in the textbook that support it.
- The word "Red" gets a line to the red patch in the image.
- The word "Above" gets a line to the top part of the image.
- The word "Not" gets a line to... well, it tells the system to look for the absence of something.

By keeping this detailed map, a small, simple computer program (a lightweight CNN) can learn to read the map. It sees the pattern: "Oh, the word 'Red' is connected to the circle patch, and the word 'Triangle' is connected to the blue patch. This matches the sentence!"

Why This Matters

It's Cheaper: You don't need to retrain the giant, expensive camera from scratch. You just take the existing camera, make a map, and run a tiny, cheap program on top of it.
It Works Better: In tests, this new method solved the "Red Circle vs. Red Triangle" problem, understood "Not a dog," and figured out "Left of" vs. "Right of" much better than the original camera.
It's Explainable: Because we are looking at the map, we can actually see why the computer made a decision. We can see exactly which word matched which part of the image.

Summary

The paper says: CLIP is a brilliant but geometrically flawed tool. It's like a Swiss Army knife that is great at cutting but terrible at screwing. You can't fix the screwdriver part just by sharpening the blade; the tool is built wrong for that specific job.

However, instead of throwing the tool away, the authors built a special adapter (the DCSM). This adapter takes the raw data from the tool, lays it out in a detailed grid, and uses a simple pattern-recognition system to interpret it correctly. This allows us to keep the speed and power of the original tool while finally fixing its confusion about order, location, and negation.

1. Problem Statement

The paper addresses fundamental geometric limitations in Contrastive Language-Image Pre-Training (CLIP) models. While CLIP is highly effective for zero-shot tasks like image classification and retrieval, it fails at complex visual-textual reasoning involving:

Attribute Binding: Distinguishing which attribute belongs to which object in multi-object scenes (e.g., "red circle blue triangle" vs. "red triangle blue circle").
Spatial Relationships: Understanding relative positions (e.g., "circle above triangle" vs. "circle below triangle").
Negation: Correctly interpreting negative constraints (e.g., "a car without a wheel").

The authors argue that these failures are not merely due to insufficient training data or suboptimal architectures but are inherent to the geometry of the CLIP latent space. They posit that the standard CLIP approach of mapping images and texts to unit vectors on a hypersphere and scoring them via cosine similarity creates an "ideal" space that is mathematically impossible to construct for these specific tasks.

2. Theoretical Analysis & Proof

The authors formalize the CLIP latent space as a projection of atomic concepts (objects, attributes, spatial relations) onto a unit hypersphere. They define four necessary conditions for an "Ideal CLIP":

Concept Categorization: Basic descriptions must match content.
Attribute Binding: Specific attribute-object pairings must be distinct.
Spatial Relationships: Different spatial arrangements must yield distinct embeddings.
Negation: Negated concepts must be semantically distinct from their positive counterparts.

Key Theoretical Finding:
The paper proves that no CLIP-like joint embedding space exists that can satisfy any two of these conditions simultaneously when using cosine similarity on a unit hypersphere.

The Superposition Problem: To satisfy Condition 1 (basic content), the embedding of a composite scene (e.g., object $x$ with attribute $a$ ) must be a linear superposition of its components.
The Contradiction: This linear superposition forces the embeddings of different attribute bindings (e.g., $x$ with $a$ vs. $y$ with $b$ ) to become indistinguishable or identical, violating Condition 2. Similarly, satisfying spatial constraints (Condition 3) or negation (Condition 4) creates geometric contradictions with the requirement to preserve basic semantic content.

Consequently, simply fine-tuning CLIP or re-projecting its embeddings cannot solve these issues because the fundamental representation (unit vectors + cosine similarity) is structurally flawed.

3. Methodology: Dense Cosine Similarity Maps (DCSMs)

To "rescue" the CLIP latent space without discarding its powerful pre-trained encoders, the authors propose Dense Cosine Similarity Maps (DCSMs).

Core Concept:
Instead of collapsing the entire image and text into single global vectors (CLS and EOS tokens) and computing a single scalar cosine similarity, the method retains the full topology of the embeddings:

Token-Patch Interaction: Compute the pairwise cosine similarity between every text token and every image patch.
DCSM Generation: This results in a 2D matrix (Dense Cosine Similarity Map) where rows represent text tokens and columns represent image patches.
Functional Rows (FRs): The authors observe that functional words (prepositions like "left," "above," negations like "no") often have inverted or noisy attention patterns in standard CLIP. They replace the rows corresponding to these functional words with constant, fixed vectors (learned or random) to act as explicit spatial/semantic anchors.
Lightweight Scorer: A simple, lightweight 2-layer Convolutional Neural Network (CNN) processes the DCSM to output a final text-image match score. The CNN learns to recognize the patterns within the dense map (e.g., high similarity clusters for correct attribute bindings) rather than relying on the flawed global vector.

Key Advantages:

Topology Preservation: The DCSM preserves the spatial arrangement of image patches and the sequential order of text tokens.
Interpretability: The maps are human-interpretable, showing exactly which text tokens align with which image regions.
Efficiency: The downstream CNN is extremely lightweight (20x fewer parameters than typical VLMs) and requires minimal training data (20k samples vs. millions for CLIP).

4. Experimental Results

The authors evaluated DCSM against state-of-the-art models (OpenCLIP, NegCLIP, CoCa, SigLIP, BLIP) on benchmarks specifically designed to test compositional reasoning:

Attribute Binding (CLEVR-bind, NCD, VG-attribution): DCSM achieved significant gains. For example, on CLEVR-bind, DCSM (trained on COCO) reached 39.9% accuracy compared to OpenCLIP's 20.2%.
Spatial Reasoning (WhatsUp, COCO-QA, VG-QA): DCSM showed massive improvements. On WhatsUp, DCSM scored 63.7% vs. CLIP's 30.5%.
Negation (NegBench): DCSM improved performance on negation tasks, though gains were slightly lower due to the gap between templated training data and natural language.
Generalization: The model generalized well to unseen concepts and single-object scenes despite being trained primarily on two-object templates, suggesting it learned syntactic patterns rather than overfitting.
Open Vocabulary: By integrating an LLM to dynamically update the Functional Row lookup table, the model showed modest improvements on open-vocabulary datasets (Sugarcrepe, VL-Checklist).

5. Significance and Contributions

Theoretical Insight: The paper provides a rigorous mathematical proof that the standard CLIP geometry is fundamentally incapable of handling compositional logic, shifting the focus from "more data" to "better geometry."
Practical Solution: It demonstrates that one can fix CLIP's reasoning flaws without retraining the massive foundation models. By utilizing the rich information in patch/token embeddings and applying a simple pattern-recognition layer (DCSM), performance on complex reasoning tasks can be doubled.
Efficiency: The proposed method is computationally cheap, requiring only a small CNN and a fraction of the training data used by CLIP, making it highly accessible for downstream applications.
Interpretability: The shift from a "black box" scalar score to a visualizable map offers new avenues for understanding and debugging vision-language models.

In summary, the paper argues that while CLIP's global geometry is broken for compositionality, its local (patch/token) information is rich and correct. The DCSM approach effectively decouples the representation from the scoring mechanism, allowing the model to leverage CLIP's features while bypassing its geometric limitations.