🤖 AI

Geometry-Aware CLIP Retrieval via Local Cross-Modal Alignment and Steering

This paper proposes a geometry-aware CLIP retrieval framework that enhances performance on attribute-binding and compositional tasks by addressing local geometric inconsistencies through query-conditioned local steering and neighborhood-level re-ranking via Hungarian matching, all achievable at inference time without retraining.

Original authors: Nirmalendu Prakash, Narmeen Fatimah Oozeer, Xin Su, Phillip Howard, Shaan Shah, Zoe Wanying He, Shuang Wu, Shivam Raval, Roy Ka-Wei Lee, Meenakshi Khosla, Amir Abdullah

Published 2026-04-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nirmalendu Prakash, Narmeen Fatimah Oozeer, Xin Su, Phillip Howard, Shaan Shah, Zoe Wanying He, Shuang Wu, Shivam Raval, Roy Ka-Wei Lee, Meenakshi Khosla, Amir Abdullah

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are at a massive, chaotic library where every book is described by a single, long sentence. You ask the librarian (the AI) for a "blue hexagon."

In a standard library, the librarian looks at the whole sentence describing the book and gives you a score based on how well the entire description matches your request. This works great if you just want "a book about shapes." But if you need a specific shape with a specific color, the librarian gets confused. They might hand you a "blue pentagon" or a "red hexagon" because, to them, those books are "close enough" in the general neighborhood of the library. They see the words "blue" and "hexagon" in the wrong places and can't tell the difference.

This paper is about teaching the librarian to stop looking at the "big picture" and start looking at the neighborhood around the books to get the details right.

Here is the breakdown of their solution using simple analogies:

1. The Problem: The "Blurry Neighborhood"

Current AI models (like CLIP) are like a person who sees a crowd of people from far away. They can tell you, "That group over there has a lot of people wearing red," but they can't tell you exactly who is wearing red and who is wearing blue.

When you ask for a "blue hexagon," the AI sees a "blue" person and a "hexagon" person standing near each other, but it might accidentally swap them. It thinks, "Hey, they are both in the red zone, so it must be the right answer!" This leads to local geometric inconsistencies: the right items are there, but they are in the wrong order or mixed up.

2. Solution A: The "Hungarian Matchmaker" (Re-ranking)

The authors realized that instead of just picking the single "best" match, we should look at the top 10 or 20 candidates and rearrange them like a puzzle.

The Analogy: Imagine you have a list of 20 potential dates. The AI initially ranks them by "overall vibe." But you realize the top pick is actually a "blue circle" and the 15th pick is the "blue hexagon" you wanted.
The Fix: The authors use a mathematical trick called Hungarian Matching. Think of this as a super-organized matchmaker who looks at your list of requirements (e.g., "I need a blue shape AND a hexagon shape") and forces the candidates to pair up perfectly. If the candidate has a blue circle and a red hexagon, the matchmaker says, "No, that's a mismatch. Let's swap you with the person who has the blue hexagon."
The Result: This instantly fixes the "top 1" answer. It's like sorting a messy deck of cards so the Ace of Spades is finally on top.

3. Solution B: The "Steering Wheel" (Local Steering)

Sometimes, the AI isn't just confused; it's stubbornly stuck on the wrong idea. Maybe it really wants to show you a "red" object, and you need it to be "blue."

The Analogy: Imagine you are driving a car (the search result) and you are slightly off-course. Instead of turning the car around completely (which would take a new engine or retraining the AI), you just gently turn the steering wheel.
The Fix: The authors found "directions" in the AI's brain. If you want to change "red" to "blue," they found a specific vector (a mathematical arrow) that points from "red" to "blue." They apply this arrow to the search query before looking for the book.
The Result: It's like telling the librarian, "I still want a hexagon, but please nudge the search slightly toward the 'blue' aisle." This allows you to control the search without rebuilding the whole library.

4. The Secret Sauce: Fused Gromov-Wasserstein (FGW)

This is the fancy math term, but the concept is simple. It's about preserving the relationships between objects.

The Analogy: Imagine you are describing a scene: "A dog is sitting next to a bicycle."
- Old AI: It hears "Dog" and "Bicycle" and thinks, "Okay, I see a dog and a bicycle somewhere." It doesn't care if the dog is on the bike or next to it.
- New AI (FGW): It looks at the geometry of the scene. It asks, "In the query, the dog and bike are neighbors. In this candidate image, are the dog and bike neighbors? If the dog is on the bike, that's a structural mismatch!"
Why it matters: This method doesn't just check if the ingredients are there; it checks if the recipe is right. It ensures that if you ask for a "blue circle next to a red star," the AI doesn't give you a "blue circle and a red star" that are far apart or swapped.

The Big Takeaway

The paper argues that we don't need to retrain the AI (which is expensive and slow) to make it smarter. We just need to change how we look at the results after the AI does its initial search.

Before: "Pick the single best match." (Often wrong for complex requests).
Now: "Look at the top 20 matches, check how the objects relate to each other, rearrange them to fit the structure, and nudge the search if needed."

By treating retrieval as a local neighborhood problem rather than a global one, they turned a confused librarian into a precise, detail-oriented expert, fixing errors like confusing a pentagon for a hexagon or mixing up colors and shapes.

1. Problem Statement

Current CLIP-based retrieval systems typically frame image-text matching as a pointwise similarity problem in a shared embedding space. While CLIP achieves strong global alignment, it suffers from local geometric inconsistencies, particularly in compositional retrieval tasks (e.g., retrieving an image with a "red pentagon" and a "blue hexagon").

The Core Issue: Standard cosine similarity collapses all objects in an image into a single global vector. This destroys the internal relational structure (spatial layout, object-to-object relationships) and leads to systematic errors where correct items exist in the top- $k$ candidates but are mis-ranked due to local neighborhood confusion (e.g., confusing a pentagon for a hexagon).
Limitations of Prior Work: Existing solutions often rely on fine-tuning the backbone or re-ranking based on global scores. These methods fail to model the local cross-modal structure and often distort fine-grained semantic distinctions when optimizing for global contrastive objectives.
Hypothesis: Retrieval failures are often local reordering problems rather than global retrieval failures. The correct item is present in the neighborhood but ranked incorrectly because the system fails to align the geometry of the local neighborhood (the set of objects and their relations) across modalities.

2. Methodology

The authors propose a two-pronged approach that operates at inference time without retraining the CLIP backbone: Local Re-ranking and Query-Conditioned Steering.

A. Local Re-ranking via Fused Gromov-Wasserstein (FGW)

Instead of treating an image as a single vector, the method decomposes both the query and the candidate images into per-object representations.

Per-Object Decomposition: Using scene annotations (or text prompts), the system generates a set of embeddings for individual objects (e.g., "large red sphere," "small blue cube") rather than a single image embedding.
Fused Gromov-Wasserstein (FGW) Optimal Transport:
- The method treats the query and candidate as two sets of objects.
- It defines a cost matrix based on feature dissimilarity (cosine distance between object embeddings) and structural dissimilarity (pairwise distances between objects within the set).
- The FGW objective minimizes a weighted sum of feature mismatch and structural distortion. It seeks a transport plan that aligns objects while preserving their internal relational geometry.
- Hard vs. Soft Assignment: The authors compare Hungarian matching (hard, one-to-one assignment) which excels at Top-1 accuracy, against FGW (soft, fractional assignment) which provides a continuous gradient of scores, significantly improving ranking quality across the entire neighborhood (R@5, R@10, nDCG).

B. Query-Conditioned Local Steering

The authors explore "steering" the retrieval space by deriving direction vectors from contrastive neighborhoods.

Mechanism: By computing the difference between text embeddings of contrasting concepts (e.g., "upright bottle" vs. "toppled bottle"), they derive a steering vector.
Application: This vector is applied to the query embedding to bias the retrieval toward specific attributes.
Finding: While steering works for single concepts, naive multi-object steering (merging multiple steered vectors into one) fails due to interference. This reinforces the need for structure-aware methods like FGW rather than vector aggregation.

3. Key Contributions

Neighborhood Alignment Framework: The paper reframes retrieval as a local cross-modal alignment problem. It demonstrates that many fine-grained failures arise from misordered local neighborhoods rather than a lack of global similarity.
FGW-Based Reranking: Introduces a two-stage reranking procedure using Fused Gromov-Wasserstein optimal transport. This restores structural consistency among candidates by comparing them as structured sets of objects, yielding massive gains in compositional retrieval without retraining.
Controllable Steering: Shows that query-conditioned concept directions can reshape local neighborhoods for attribute correction, but highlights that this requires structure-aware matching to avoid interference in multi-object scenarios.
Comprehensive Evaluation: Validates the approach across synthetic (Synthetic Shapes, CLEVR-HOPE) and real-world datasets (Urban1K, Visual Genome, SugarCrepe), showing that preserving object-level structure is superior to collapsing it into a single vector.

4. Results

The proposed methods demonstrate significant improvements over baseline CLIP and standard reranking techniques:

Synthetic Shapes (Compositional):
- Baseline CLIP R@1: 3.5%
- Local FGW R@1: 5.1% (modest gain)
- Ridge + Local FGW R@1: 100.0% (Perfect retrieval when combined with a simple linear mapper).
- Note: The paper highlights that while FGW alone helps, the combination with structural alignment is key.
Urban1K (Long-caption):
- Baseline CLIP R@1: 42.8%
- Local Hungarian R@1: 78.4%
- Local FGW R@1: 56.3% (Hungarian outperforms FGW here for Top-1, likely due to the discrete nature of the task).
Compositional Retrieval (CLEVR-HOPE & Visual Genome):
- CLEVR-HOPE: FGW reranking improved R@10 from 0.230 (Baseline) to 0.682. Naive multi-object steering degraded performance to 0.094.
- Visual Genome: FGW reranking improved R@1 from 0.091 to 0.725 (using Hungarian reranking on the FGW shortlist).
Key Insight: FGW yields the largest gains in R@5, R@10, and nDCG, proving that structural matching preserves relational geometry across the neighborhood, not just the top-1 match.

5. Significance

Inference-Time Efficiency: The method improves retrieval quality without retraining the massive CLIP backbone. It operates as a post-hoc processing step, making it highly practical for deployment.
Shift in Paradigm: It challenges the dominance of global cosine similarity, arguing that local neighborhood structure is a more stable and semantically meaningful signal for retrieval than global distances.
Solving Compositional Reasoning: It provides a robust solution to the "binding problem" in vision-language models (correctly associating attributes with specific objects), a known weakness of current foundation models.
Controllability: It demonstrates that retrieval systems can be steered along interpretable semantic directions, offering a path toward more controllable and precise image search systems.

In conclusion, the paper establishes that local geometric consistency is the critical missing link in CLIP retrieval. By leveraging optimal transport to align the internal structure of object sets, the authors achieve state-of-the-art performance in compositional and attribute-binding tasks.