💻 computer science

Grad-ECLIP: Gradient-based Visual and Textual Explanations for CLIP

The paper proposes Grad-ECLIP, a gradient-based method that generates high-quality visual and textual explanations for CLIP by leveraging channel and spatial weights on token features rather than sparse self-attention maps, thereby revealing the model's matching mechanisms and enabling improved fine-grained alignment during fine-tuning.

Original authors: Chenyang Zhao, Kun Wang, Janet H. Hsiao, Antoni B. Chan

Published 2026-08-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chenyang Zhao, Kun Wang, Janet H. Hsiao, Antoni B. Chan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern world of artificial intelligence, there exists a class of systems known as vision-language models. These are digital minds trained on vast collections of images and the text that describes them, learning to understand how a picture of a dog relates to the word "dog." They have become powerful tools, capable of finding specific images in massive databases or answering questions about what they see. However, a significant mystery has surrounded these systems: while they produce correct answers, no one truly knows which parts of an image or which specific words in a sentence are driving those decisions. It is like watching a brilliant student solve a complex math problem perfectly, yet being unable to see the steps they took to get there. Without this insight, it is difficult to know if the system is truly understanding the world or simply guessing based on hidden patterns.

Researchers have long tried to peek inside these models to see what they are focusing on. Some methods look at the internal "attention" mechanisms, which are designed to highlight important information, but in these specific models, those highlights often turn out to be sparse and confusing, pointing to random spots rather than the actual subject. Other approaches try to measure how much the answer changes when parts of the image are removed, but these methods can be slow or produce noisy, unclear results. The core challenge has been finding a way to clearly show, for any given image and sentence, exactly which pixels and which words matter most to the model's final decision.

A team of researchers has now introduced a new method called Grad-ECLIP to solve this problem. Instead of relying on the model's internal attention maps, which often fail to show the full picture, their approach calculates the importance of every part of the image and every word in the text by measuring how sensitive the final result is to small changes. Imagine the model's decision as a delicate balance; this method gently nudges different parts of the input to see which ones cause the biggest shift in the outcome. If removing a specific patch of pixels or a single word causes the model's confidence to drop significantly, that part is marked as highly important. By doing this, the researchers can generate clear heat maps that glow over the most relevant areas of an image and highlight the most critical words in a sentence.

When tested against existing techniques, this new method proved to be far more accurate. In visual tests, while older methods often highlighted background noise or unrelated objects, Grad-ECLIP consistently focused on the correct subjects. For example, when matching an image of a dog playing with a frisbee to the sentence "a dog is playing with frisbee," the method correctly highlighted the dog and the frisbee, while ignoring the background. It also successfully identified that the word "playing" corresponded to the action of the dog's legs, and "frisbee" matched the object in the air. In quantitative tests, where the researchers systematically deleted or added pixels based on the generated maps, the new method caused the model's performance to change more drastically than any other technique, proving that it was indeed identifying the most crucial information.

Beyond simply showing how the model works, the researchers used this tool to uncover how the model actually thinks. They discovered that the system has a remarkable ability to break down complex phrases into individual concepts and then combine them. If shown a picture of a horse and asked about a "young horse," the model would focus on the horse but strengthen its attention on the younger one. However, if asked about a "white horse" when the horse in the picture was brown, the model would largely ignore the color and focus on the horse itself, suggesting it treats the mismatched adjective as a secondary detail rather than a dealbreaker. This revealed that the model tends to prioritize concrete, visual concepts like colors and shapes over abstract comparisons like "left" or "right," which it often struggles to locate precisely.

The study also found that the model places much higher importance on concrete words—those that describe things you can see and touch—compared to abstract words. This preference is not simply because concrete words appear more often in the training data; the researchers checked the frequency of words and found no direct link. Instead, the model seems to have learned that concrete visual details are more reliable for matching images to text. This insight helps explain why these systems are so good at zero-shot learning, where they can recognize new things they have never seen before, by relying on the solid, visual building blocks they have mastered.

Ultimately, this work provides a clear window into the reasoning of a complex artificial intelligence. By showing exactly where the model looks and what it values, the researchers have moved beyond treating these systems as black boxes. The new method allows scientists and developers to trust the model's decisions more confidently, knowing exactly which features led to a conclusion. It also highlights the model's current limitations, such as its difficulty with spatial relationships, offering a roadmap for future improvements. With this tool, the path forward for building more reliable and understandable artificial intelligence becomes much clearer.

Technical Summary of Grad-ECLIP: Gradient-based Visual and Textual Explanations for CLIP

Problem Statement
While Contrastive Language-Image Pre-training (CLIP) has achieved significant success in downstream applications, the interpretability of its image-text matching mechanism remains under-explored. Existing explanation methods for Transformer-based models often rely on self-attention maps, which are typically extremely sparse in CLIP, leading to low-faithfulness and confusing visualizations. Other approaches, such as similarity-map methods (e.g., CLIPSurgery, MaskCLIP), rely solely on forward processing and may highlight features not actually used in the final prediction. Furthermore, gradient-based methods like Grad-CAM struggle when applied directly to CLIP's Vision Transformer (ViT) layers because the final patch tokens are not involved in the matching score calculation. There is a need for a method that effectively decomposes the encoder architecture to generate high-fidelity, result-specific visual and textual explanations for both image and text inputs.

Methodology: Grad-ECLIP
The authors propose Grad-ECLIP, a gradient-based explanation method tailored for the dual-encoder architecture of CLIP. The method derives a relationship between the final image-text similarity score and intermediate spatial features through a series of approximations.

Decomposition and Approximation: The method approximates the final image feature embedding ( $F_I$ ) as an aggregation of features from each transformer layer. It further approximates the matching score as a weighted sum of channel features in the attention layer's output ( $o_{cls}$ ).
Gradient-Based Channel Weights: To determine the importance of feature channels, the method calculates the gradient of the image-text matching score with respect to the attention layer output ( $o_{cls}$ ). These gradients serve as channel weights ( $w_c$ ).
Loosened Spatial Weights: Recognizing that standard softmax attention maps are too sparse to effectively represent spatial importance, the authors propose a "loosened" attention map. Instead of using the strict softmax output, they apply 0-1 normalization to the similarity scores between the query and keys ( $q_{cls}k_i^T$ ) to generate spatial weights ( $\lambda_i$ ). This allows the method to reflect the importance of a broader range of regions rather than just the single most activated token.
Heat Map Generation: The final explanation map is generated by aggregating the value features ( $v_i$ ) from the attention layer, weighted by the channel importance ( $w_c$ ) and the loosened spatial importance ( $\lambda_i$ ). A ReLU function is applied to focus only on features with a positive impact on the score.
Applicability: The same framework is applied to the text encoder (using the [eos] token) to generate textual explanations, highlighting the importance of specific words in a sentence relative to an image. The method is applicable to both ViT-based and CNN-based (ResNet with attention pooling) CLIP architectures.

Key Contributions

Novel Explanation Framework: The paper introduces Grad-ECLIP, a gradient-based approach that produces high-quality, result-specific heat maps for both image regions and text words in CLIP models.
Superiority over State-of-the-Art: Through comprehensive qualitative and quantitative evaluations, Grad-ECLIP is shown to outperform existing Transformer interpretation methods (e.g., Rollout, GAME) and CLIP-specific methods (e.g., CLIPSurgery, MaskCLIP, M2IB) in terms of faithfulness.
Generalizability: The method is validated across diverse datasets (ImageNet, MS COCO, Conceptual Captions, medical X-rays) and different domains (sketches, adversarial examples). It is also demonstrated to be applicable to other Transformer-based models, including ViT classifiers and BLIP, as well as CNN-based CLIP variants.
Insights into CLIP Mechanisms: Using Grad-ECLIP, the authors analyze the internal workings of CLIP, revealing:
- Concept Decomposition and Addibility: CLIP processes phrases by decomposing them into individual words and adding their contributions, rather than treating phrases as monolithic units.
- Attribute Identification: CLIP shows strong capabilities in identifying concrete attributes like color and shape but struggles with comparative attributes (size, position) and abstract concepts.
- Concreteness Bias: There is a significant positive correlation between word concreteness and word importance in CLIP's matching process, a bias that persists even when controlling for word frequency.

Experimental Results

Quantitative Evaluation: Grad-ECLIP achieves the highest Insertion Minus Deletion (IMD) scores on both ImageNet classification and MS COCO retrieval tasks, indicating that the highlighted regions are most critical to the model's prediction. It significantly outperforms baselines in Deletion and Insertion metrics ( $p < .001$ ).
Localization: In Point Game and segmentation tests, Grad-ECLIP demonstrates superior localization accuracy, correctly identifying object regions with higher precision than competing methods.
User Study: A user study involving 34 participants showed that providing Grad-ECLIP explanations significantly improved users' ability to predict CLIP's matching scores (accuracy increased from 67.6% to 80.0%), validating the trustworthiness and interpretability of the method.
Efficiency: The method is computationally efficient, with processing times comparable to Grad-CAM and raw attention, and significantly faster than perturbation-based methods like RISE.

Significance and Claims
The paper claims that Grad-ECLIP provides a robust tool for understanding the "black box" nature of CLIP. By revealing that CLIP relies heavily on concrete visual concepts and decomposes phrases additively, the method offers developers and researchers insights into the model's strengths and limitations. The authors suggest that these findings can guide the improvement of Vision-Language Models (VLMs) and their downstream applications. Additionally, the paper demonstrates a practical application where Grad-ECLIP is used to generate region-phrase pairs to fine-tune CLIP for better fine-grained understanding, showing effectiveness in zero-shot region classification and open-vocabulary detection. The work positions itself as a step toward more transparent and interpretable multi-modal AI systems.

More like this