CLAY: Conditional Visual Similarity Modulation in Vision-Language Embedding Space

The paper proposes CLAY, a training-free method that leverages pretrained Vision-Language Models to enable adaptive, multi-conditioned visual similarity retrieval by reframing the embedding space as text-conditional, achieving high accuracy and efficiency without requiring additional model training.

The Gist

Imagine you are looking for a specific photo in a massive, chaotic digital photo album containing millions of pictures.

The Problem: The "One-Size-Fits-All" Search
Most image search engines today work like a very strict librarian who only cares about one thing: "How much does this picture look like that one?"

• If you show them a picture of a red car, they will find other red cars.
• But what if you say, "I don't care about the color; I just want to see cars driving in the rain"?
• Or, "I want to see dogs, but specifically dogs wearing hats"?

Current systems struggle with this. They are like a rigid robot that can't shift its focus. If you ask for a "red car," it ignores the fact that you might actually care more about the action of driving. To change the focus, you usually have to retrain the whole system or start over, which is slow and expensive.

The Solution: CLAY (The "Smart Lens")
The authors of this paper, from KAIST, created a new method called CLAY. Think of CLAY as a smart, adjustable lens you can slide over your photo album.

Instead of changing the photos themselves, CLAY changes how you look at them based on what you are interested in at that moment.

Here is how it works, using a creative analogy:

1. The "Universal Photo Album" (The Pre-trained Model)

Imagine you already have a giant, perfect photo album where every picture is organized by a super-smart AI (called a Vision-Language Model, or VLM). This AI knows that a "dog" looks like a "dog" and a "cat" looks like a "cat." It doesn't need to be taught; it already knows everything.

2. The "Magic Filter" (The Conditional Space)

Now, imagine you want to find pictures of dogs running.

• Old Way: You have to go through the whole album, pick out every dog, check if it's running, and throw the rest away. This takes forever.
• CLAY Way: You slide a "Running" filter over the lens. Suddenly, the entire album rearranges itself. The pictures of dogs running move to the front, and the pictures of dogs sleeping or cats running get pushed to the back.
• The Magic: CLAY does this rearranging instantly without touching the original photos. It just changes the "similarity rules" for that specific moment.

3. The "Geometry Trick" (How it actually works)

The paper explains that the AI's brain (the "embedding space") is shaped like a sphere, not a flat sheet of paper.

• The Problem: If you try to draw a straight line on a sphere, it gets distorted. Old methods tried to draw straight lines on a curved ball, which made the results messy.
• The CLAY Fix: CLAY uses a "mapmaker's trick." It flattens the specific part of the sphere you are interested in (like unrolling a globe to look at just one continent) so it can compare things fairly. It then uses a "rotation" to make sure the "Running" filter aligns perfectly with the "Dog" pictures.

4. The "No-Training" Superpower

Usually, teaching a computer to understand "running dogs" vs. "sleeping dogs" requires showing it thousands of examples and waiting days for it to learn.

• CLAY is Training-Free: It doesn't need to learn anything new. It takes the knowledge the AI already has and simply re-organizes it based on your text instruction.
• Analogy: It's like having a library where the books are already sorted alphabetically. If you want to find books about "Space," you don't need to rewrite the books; you just put a sign on the shelf that says "Focus on Space," and the librarian instantly knows to show you those books first.

Why This Matters

• Speed: Because it doesn't need to re-calculate the photos every time you change your mind, it is incredibly fast. You can switch from "Find me red cars" to "Find me fast cars" in a blink.
• Flexibility: You can mix conditions. "Find me old people reading in a park." CLAY can handle all three conditions at once.
• Real-World Use: The authors even built a fake (synthetic) photo album called CLAY-EVAL to test this, proving it works on everything from animals to people doing different actions.

In a Nutshell:
CLAY is like giving your photo search engine a chameleon's ability. It doesn't change the photos; it changes the perspective instantly to match exactly what your human brain is interested in at that second, making the search feel natural, flexible, and lightning-fast.

Technical Summary

1. Problem Definition

Current image retrieval systems typically rely on static, monolithic similarity metrics (e.g., fixed cosine similarity in a pre-trained embedding space). While effective for general tasks, they fail to capture the adaptive and subjective nature of human visual perception. Humans often seek images based on specific, changing criteria (e.g., "find similar dogs" vs. "find similar locations" vs. "find similar actions") without altering the query image itself.

Existing solutions for Conditional Image Retrieval suffer from two main limitations:

1. Training Dependency: Most methods require training on specific condition-target pairs, limiting them to closed-set conditions and requiring retraining for new attributes.
2. Computational Inefficiency: Symmetric approaches (where both query and database images are re-encoded with the condition) incur massive computational overhead because database features must be recomputed for every new condition. Asymmetric approaches (conditioning only the query) are efficient but often yield suboptimal results because the database features remain "condition-agnostic."

2. Methodology: CLAY

The authors propose CLAY, a training-free method that adaptively modulates the similarity space of pre-trained Vision-Language Models (VLMs) like CLIP or SigLIP. The core innovation is decoupling the visual feature extraction from the textual conditioning process.

Key Technical Components:

• Decoupled Conditioning: CLAY keeps the visual embeddings of the database images fixed (pre-computed). Instead of re-encoding the database for every new condition, it modulates the similarity computation space itself.
• Manifold-Aware Textual Subspace Construction:
  • VLM embeddings lie on a unit hypersphere, not a Euclidean space. Standard linear subspaces fail to capture the intrinsic geometry.
  • CLAY constructs a textual subspace for a given condition $c$ by:
    1. Generating diverse text prompts related to the condition using an LLM (e.g., "a photo of running," "a photo of jumping").
    2. Mapping these text embeddings from the hypersphere to a local tangent space using a logarithm map centered at the mean of the text features ($\mu_c$).
    3. Applying Singular Value Decomposition (SVD) on the mapped features to extract the top-$k$ singular vectors, forming a projection matrix $P_c$.
• Inference Pipeline (Symmetric Modulation without Re-encoding):
  1. Alignment: To handle the "conic effect" (where visual features are far from the text mean), CLAY applies an orthonormal rotation $H(\cdot)$ to align the mean of the database visual features with the text mean $\mu_c$.
  2. Projection: Both the query and database visual features are mapped to the tangent space of $\mu_c$, rotated, and then projected onto the condition-specific textual subspace using $P_c$.
  3. Similarity: The final similarity is computed via cosine similarity in this modulated space.
  • Result: This allows for symmetric conditioning (modulating both query and DB) with the computational cost of an asymmetric approach, as the database projection is a simple linear operation ($P_c \cdot v$) rather than a full forward pass through a neural network.

3. Key Contributions

1. CLAY Algorithm: A novel, training-free, and efficient method for conditional visual similarity that achieves state-of-the-art performance without retraining or re-encoding database images.
2. Multi-Condition Support: Unlike previous methods restricted to single conditions, CLAY naturally extends to multi-conditioned retrieval (e.g., "red car" AND "in the city") by constructing textual subspaces from combined prompts.
3. CLAY-EVAL Dataset: The authors introduce a large-scale synthetic evaluation dataset containing ~14,000 images (objects and humans) with diverse, disentangled attributes (e.g., species, action, color, background). This dataset addresses the lack of standardized benchmarks for multi-conditional retrieval.
4. Geometric Insight: The paper demonstrates that respecting the hyperspherical geometry of VLM embeddings via tangent space approximation is crucial for accurate relationship modeling.

4. Experimental Results

The method was evaluated on real-world fine-grained datasets (Stanford40, OxfordPets, Food-101, etc.) and the synthetic CLAY-EVAL dataset.

• Performance: CLAY consistently outperforms baselines (CLIP, SigLIP, GeneCIS, FocalLens, InstructBLIP) across all datasets.
  • On Stanford40 (Action condition), CLAY (SigLIP-B) achieved 66.2 mAP, significantly beating the baseline SigLIP (54.8) and GeneCIS (50.9).
  • On CLAY-EVAL, it showed massive gains, e.g., 94.3 mAP on the "Category" condition for objects, compared to 69.6 for SigLIP-B.
• Efficiency:
  • Training-free: No fine-tuning required.
  • Inference Speed: CLAY maintains high efficiency. While symmetric methods (like modified GeneCIS) require re-encoding the entire database (taking 1.7s per query for the second condition), CLAY only requires a projection step (0.09s), making it practical for large-scale databases.
• Multi-Condition: CLAY is the only method tested that effectively handles multi-condition queries (e.g., Age + Action + Background) without performance collapse, achieving 81.5 mAP on the "All" condition for CLAY-Human.
• Qualitative Analysis: t-SNE visualizations show that CLAY creates highly discriminative clusters aligned with specific conditions (e.g., separating "running" from "reading" humans), whereas baseline CLIP features remain mixed.

5. Significance and Impact

• Bridging the Gap: CLAY successfully bridges the gap between accuracy (usually requiring heavy training/re-encoding) and efficiency (usually requiring static embeddings). It proves that conditional retrieval can be achieved by manipulating the metric space rather than the features themselves.
• Practical Applicability: By eliminating the need for re-encoding database images, CLAY makes adaptive, user-intent-driven retrieval feasible for massive, real-world image repositories.
• Future Directions: The work opens avenues for applying conditional similarity modulation to other multimodal tasks, such as text-to-image generation alignment and fine-grained visual matching in generative models.

In summary, CLAY represents a paradigm shift in image retrieval, moving from static similarity metrics to dynamic, text-conditioned similarity spaces that are both geometrically sound and computationally efficient.