💬 NLP

Revisiting Compositionality in Dual-Encoder Vision-Language Models: The Role of Inference

This paper argues that the poor compositional performance of dual-encoder Vision-Language Models stems from their reliance on global cosine similarity during inference, demonstrating that introducing lightweight, fine-grained region-segment alignment mechanisms over frozen representations significantly enhances compositional generalization and robustness to distribution shifts compared to full fine-tuning.

Original authors: Imanol Miranda, Ander Salaberria, Eneko Agirre, Gorka Azkune

Published 2026-04-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Imanol Miranda, Ander Salaberria, Eneko Agirre, Gorka Azkune

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Bag of Words" Blind Spot

Imagine you have a very smart librarian (the AI model) who has read millions of books and looked at millions of pictures. This librarian is great at matching a picture of a dog with the word "dog."

However, this librarian has a weird blind spot. If you show them a picture of a black dog and a white cat, and ask them to match it with the text "a black cat and a white dog," they often get it wrong. They see the words "black," "white," "dog," and "cat," and they think, "Hey, all those words are in the picture! It's a match!"

They fail to notice who is wearing what. They treat the image like a "bag of words" where the order and relationships don't matter, just the ingredients. In the AI world, this is called a lack of compositional reasoning.

The Old Theory: "The Librarian is Dumb"

For a long time, researchers thought the problem was the librarian's brain (the model's internal memory). They thought, "The librarian just hasn't learned enough about how objects relate to each other. We need to retrain their brain to understand that 'black' belongs to the 'dog,' not the 'cat'."

So, they tried to retrain these models with harder examples, hoping to force the brain to learn these connections.

The New Discovery: "The Librarian is Smart, But the Test is Flawed"

This paper argues something different. The authors say: "The librarian actually knows the difference! The problem is how we are asking them the question."

In standard AI testing, we ask the librarian to compare the "whole picture" to the "whole sentence" using a single, giant score (called Global Cosine Similarity). It's like asking the librarian to compare two entire libraries by looking at the average color of the books on the shelves. It's too broad. It misses the specific details.

The authors propose that the "brain" (the frozen encoder) already contains the fine-grained details (the specific patches of the image and the specific words), but the standard testing method ignores them.

The Solution: Two New Ways to Test

The authors tested two new ways to help the librarian use the information they already have.

1. The "Spotlight" Method (Structure-Guided Inference)

Imagine instead of looking at the whole library at once, you shine a flashlight on specific parts.

Step 1: You take the sentence "A black dog and a white cat" and break it into pieces: "black dog" and "white cat."
Step 2: You take the picture and cut it into little squares (patches).
Step 3: You force the librarian to match "black dog" specifically to the part of the picture with the black dog, and "white cat" to the white cat.
Result: When you do this, the librarian suddenly gets the answer right! They didn't need new training; they just needed to be asked to look at the details.

2. The "Translator" Method (Learning Alignment)

The authors then built a tiny, lightweight "translator" (a small neural network) that sits on top of the frozen librarian.

This translator's only job is to learn how to connect the specific image patches to the specific words.
It doesn't change the librarian's brain; it just teaches the librarian how to organize the information they already have.
Result: This tiny translator performed better than retraining the whole librarian from scratch, especially when the librarian faced new, tricky situations (out-of-distribution data).

The Key Takeaways (The "Aha!" Moments)

The Brain is Fine: The pre-trained models (like CLIP) actually do have the information needed to tell a black dog from a black cat. They aren't "dumb"; they are just being tested with a blunt instrument.
The Test is the Bottleneck: The standard way of comparing images and text (global similarity) is the main reason these models fail at composition. It's like trying to solve a puzzle by squinting at the whole box instead of looking at the individual pieces.
Small Changes, Big Gains: You don't need to retrain the massive, expensive AI models. You just need a small, smart mechanism to help them align the specific parts of the image with the specific parts of the text.
Robustness: When the models were tested on completely new types of puzzles (out-of-distribution), the ones using this new "alignment" method worked much better than the ones that were fully retrained.

The Analogy Summary

The Old Way: Trying to identify a specific person in a crowd by looking at the average skin tone of the whole crowd. (Fails miserably).
The Paper's Way: Giving the observer a magnifying glass and a checklist to find the specific person's face, hat, and shoes, then matching those specific details to the description. (Works perfectly).

In short: The paper suggests that to make AI smarter at understanding complex relationships, we shouldn't just make the AI bigger or retrain it harder. Instead, we should change how we ask it to think, forcing it to pay attention to the specific connections between words and image parts.

1. Problem Statement

Dual-encoder Vision-Language Models (VLMs), such as CLIP and SigLIP, have achieved state-of-the-art results in image-text retrieval and zero-shot recognition. However, they consistently struggle with compositional reasoning—the ability to correctly bind objects, attributes, and relations (e.g., distinguishing "a black dog and a white cat" from "a black cat and a white dog").

Current Consensus: The community often attributes these failures to deficient representations, assuming the models lack the capacity to encode fine-grained compositional structures.
Authors' Hypothesis: The paper argues that the limitation stems not from the learned representations themselves, but from the standard inference protocol. Standard inference relies on global cosine similarity between pooled image and text embeddings. This mechanism collapses multimodal information into a single vector, discarding the region-to-token correspondences necessary for compositional binding.

2. Methodology

The authors employ a two-stage approach: a diagnostic study to isolate the inference bottleneck, followed by a learning-based approach to validate the hypothesis.

A. Diagnostic Study: Structure-Guided Inference (SGI)

To test if better alignment improves performance without retraining, the authors designed a controlled inference protocol:

Freezing Encoders: Pretrained VLM encoders (CLIP, SigLIP 2, Perception Encoder) remain frozen.
Decomposition:
- Visual: Images are decomposed into a set of crops (approximating object regions) using predefined scales and strides.
- Text: Captions are deterministically parsed into structured segments (e.g., "black cat", "white dog").
Local Matching: Instead of a global dot product, the system computes cosine similarity between every image crop and every text segment.
Aggregation: For each text segment, the crop with the highest similarity is selected. The final score is the average of these maximum similarities.
Benchmark: They introduce BISCOR-CTRL, a controlled out-of-distribution (OOD) benchmark based on CLEVR synthetic scenes. It focuses exclusively on SWAP cases (swapping attributes/objects) to eliminate spurious correlations and force true compositional binding.

B. Learning-Based Approach: Alignment Transformer

Building on the diagnostic results, the authors propose a lightweight method to learn these alignments directly from frozen representations:

Architecture: A lightweight Transformer module operates on top of frozen patch (visual) and token (text) embeddings.
Mechanism: It uses cross-modal attention to learn fine-grained correspondences between image patches and text tokens.
Training: Only the parameters of this lightweight transformer are updated (using a contrastive objective); the backbone VLM encoders remain frozen.
Comparisons: The method is compared against:
1. Full Fine-Tuning (FT): Updating all encoder parameters.
2. Global Transformer (TFGlobal): Applying the same transformer architecture but only on global pooled embeddings (to test if capacity alone is the issue).
3. Prior End-to-End Methods: Models trained with compositional objectives (e.g., NegCLIP, FSC-CLIP) that still use global cosine inference.

3. Key Contributions

Identification of the Bottleneck: The paper demonstrates that global cosine similarity is the primary bottleneck for compositional reasoning in dual-encoder VLMs, not the representational capacity of the encoders.
Diagnostic Evidence: They show that enforcing structured region-segment alignment at inference time (SGI) on frozen models dramatically improves compositional performance, proving that the necessary information already exists in the pretrained embeddings.
Novel Method (TFLocal): They introduce a lightweight alignment transformer that learns to exploit patch/token embeddings. This method matches or exceeds full fine-tuning on in-domain tasks while significantly outperforming it on OOD tasks.
New Benchmark (BISCOR-CTRL): A rigorous, controlled OOD benchmark designed to isolate compositional reasoning from dataset biases and spurious correlations.

4. Key Results

Diagnostic Results (SGI on Frozen Models)

Performance Leap: Applying SGI to frozen models increased average group scores on BISCOR-CTRL from near-zero (1–9 points) to substantial levels (25–56 points).
Implication: The pretrained models contain rich compositional information, but the standard inference protocol fails to access it.

Learning Results (Alignment Transformer vs. Fine-Tuning)

In-Domain Performance: The lightweight alignment transformer (TFLocal) matches or slightly exceeds Full Fine-Tuning (FT) on standard benchmarks (SUGARCREPE, BIVLC).
Out-of-Domain (OOD) Generalization:
- FT Failure: Full fine-tuning showed negligible improvement on BISCOR-CTRL (e.g., CLIP: 1.4 $\to$ 1.9).
- TFLocal Success: The alignment transformer yielded massive gains on BISCOR-CTRL (e.g., CLIP: 1.4 $\to$ 15.1; PE: 8.5 $\to$ 30.0).
Global vs. Local: Applying a transformer to global embeddings (TFGlobal) failed to improve OOD performance, confirming that modeling capacity alone is insufficient; local alignment is the critical factor.
Comparison with SOTA: Existing end-to-end compositional training methods (e.g., NegCLIP, TripletCLIP) improved in-domain scores but failed to transfer to BISCOR-CTRL, whereas the proposed alignment learning method succeeded.

5. Significance and Conclusion

Paradigm Shift: The paper challenges the prevailing view that VLMs need larger or differently trained encoders to solve compositionality. Instead, it argues for revisiting inference protocols.
Efficiency: The proposed solution (learning alignment on frozen encoders) is computationally efficient, requiring only a small fraction of parameters compared to full fine-tuning.
Robustness: Localized alignment mechanisms provide superior robustness under distribution shifts compared to global matching, suggesting that future VLM architectures should prioritize mechanisms that preserve and utilize fine-grained region-token correspondences during inference.
Practical Impact: This work suggests that significant gains in compositional reasoning can be achieved without retraining massive foundation models, simply by changing how their outputs are aggregated and matched.

In summary, the authors conclude that compositional failures in dual-encoder VLMs are primarily an inference problem, not a representation problem. By shifting from global matching to localized alignment, the latent compositional capabilities of existing models can be unlocked.