Visual-TCAV: Concept-based Attribution and Saliency Maps for Post-hoc Explainability in Image Classification

Technical Summary: Visual-TCAV

Problem Statement
Convolutional Neural Networks (CNNs) achieve high performance in image classification but lack interpretability, often functioning as "black-box" models. This opacity reduces trust in critical domains like healthcare and autonomous driving and hinders debugging. Existing Explainable AI (XAI) methods fall into two categories with distinct limitations:

1. Saliency Methods (e.g., Grad-CAM, Integrated Gradients): These provide local, pixel-level explanations highlighting regions relevant to a specific class prediction. However, they cannot explain which high-level semantic concepts (e.g., "dimples" vs. "spherical shape") drove the prediction.
2. Concept-Based Methods (e.g., TCAV): These assess whether a network is sensitive to human-defined concepts (e.g., "striped") by computing Concept Activation Vectors (CAVs). However, TCAV provides only global, class-level sensitivity scores. It cannot localize where a concept appears in a specific input image, nor can it quantify the magnitude of a concept’s contribution to a specific prediction. TCAV scores rely on the signs of directional derivatives, meaning they cannot distinguish between concepts that have equal sensitivity but different magnitudes of importance.

Methodology: Visual-TCAV
The authors propose Visual-TCAV, a framework that integrates saliency and concept-based methods to provide both local and global explanations. The method operates on any CNN layer producing feature maps and satisfies three desiderata: spatial grounding, per-instance attribution, and aggregatability.

The pipeline consists of three main components:

1. CAV Generation and Spatial Pooling:

   • Following the TCAV framework, a Concept Activation Vector (CAV) is computed using the "Difference of Means" method on feature maps from positive concept examples and negative random examples.
   • To make the CAV location-agnostic, a Global Average Pooling (GAP) operation is applied, resulting in a pooled-CAV. This vector represents the correlation between each feature map channel and the concept.

2. Concept Map (Spatial Grounding):

   • A concept map $M^c$ is generated by computing a weighted sum of the input image’s feature maps, where weights are derived from the pooled-CAV.
   • A ReLU function retains only positively correlated regions.
   • The map is normalized using bounds derived from the median contraharmonic means of concept maps from positive and negative example sets. This produces a class-agnostic visualization showing where the network recognizes the concept.

3. Concept Attribution (Per-Instance Importance):

   • Integrated Gradients (IG) in Feature Space: Instead of applying IG to input pixels, the authors apply it to the feature maps of a chosen layer. This computes attributions for each feature map channel with respect to the target class logit.
   • Normalization: Attributions are normalized so their sum equals the normalized difference between the input and baseline logits, retaining only positive contributions.
   • Weighting and Masking: The normalized IG attributions are weighted by the normalized pooled-CAV (using ReLU and max normalization) to emphasize feature maps aligned with the concept. This weighted attribution map is then element-wise multiplied by the normalized concept map (acting as a spatial mask).
   • Final Score: The resulting tensor is summed over all spatial locations to yield a scalar concept attribution score $a_{c,k}$, quantifying the importance of concept $c$ for class $k$ for that specific input.

Key Contributions

• Bridging Local and Global Explanations: Visual-TCAV provides class-agnostic saliency maps (concept maps) that localize concepts and per-instance attribution scores that quantify concept importance.
• Magnitude-Based Attribution: Unlike TCAV, which relies on gradient signs, Visual-TCAV uses Integrated Gradients to capture the magnitude of concept influence, allowing for differentiation between concepts with similar sensitivity but different impact.
• Global Aggregation: Per-instance attributions can be averaged across multiple inputs to derive global, class-level explanations.

Results
The authors evaluate Visual-TCAV on ResNet50V2, InceptionV3, and VGG16 (ImageNet) and a ResNet50V2 trained on CelebA (gender classification).

1. Local Explanations: Concept maps successfully localize concepts (e.g., "car," "pews") across different layers. Deeper layers show higher attributions for class-discriminative concepts, aligning with known CNN behavior. The method reveals causes of mispredictions; for example, an "ox" misclassified as a "dalmatian" was driven primarily by the "spotted" concept.
2. Global Explanations: Aggregated attributions align with intuitive expectations (e.g., "striped" is highly important for "zebra"). The results also highlight potential model biases, such as over-reliance on "lipstick" for gender classification.
3. Quantitative Validation (Ground Truth Experiment):
   • The authors trained models on datasets with varying percentages of images containing artificial tags (e.g., "Z" for zebra).
   • Ground truth importance was determined by testing models on incorrectly tagged images.
   • Visual-TCAV attributions closely mirrored the ground truth: as tag prevalence increased, the attribution for the tag increased while entity attribution decreased, correlating with model accuracy drops.
   • Comparison with TCAV: TCAV scores failed to capture these variations, showing high sensitivity to both entities and tags regardless of their actual predictive importance. This confirms that TCAV’s reliance on gradient signs is insufficient for measuring relative concept importance, whereas Visual-TCAV’s magnitude-based approach is more faithful.

Significance and Limitations
The paper claims Visual-TCAV offers a more comprehensive explanation than TCAV by quantifying the extent of concept influence rather than just detecting sensitivity. It provides visual evidence that CAVs align with intended concepts via concept maps.

Limitations noted by the authors:

• Concept Collection: Requires manual effort and domain knowledge to gather example images, introducing potential subjectivity.
• Non-Additivity: Concept attributions are not additive; thus, the method does not provide a completeness guarantee over an arbitrary set of concepts.
• Architecture Scope: The current evaluation focuses on CNNs. Application to Vision Transformers (ViTs) is left for future work due to challenges in spatial grounding caused by self-attention token mixing.