Measuring the (Un)Faithfulness of Concept-Based Explanations

1. Problem Statement

Deep learning models, particularly in vision tasks, are often "black boxes." While Concept-Based Explanation Methods (CBEMs) attempt to bridge this gap by mapping model computations to human-understandable semantic units (concepts like "wheels" or "stripes"), a critical issue remains: faithfulness.

• The Trade-off: There is an inherent tension between interpretability (making the model understandable to humans) and faithfulness (ensuring the explanation accurately reflects the model's internal logic).
• The Gap: Existing Unsupervised CBEMs (U-CBEMs) claim to be both highly interpretable and faithful. However, the authors argue that current evaluation metrics are flawed, leading to a false consensus that these methods are faithful.
• Specific Flaws in Current Metrics:
  1. Overly Complex Surrogates: Current methods use complex, non-linear surrogates (e.g., multi-layer perceptrons or reconstruction networks) to map concepts back to model outputs. This shifts the interpretability burden downstream; if the surrogate is complex, the explanation is no longer truly interpretable by a human.
  2. Deletion-Based Proxies: Many methods rely on "deletion" (removing concepts and measuring performance drop). The authors argue this is unreliable because deleting concepts often moves inputs off the data manifold (creating unrealistic images), causing unpredictable model behavior that does not reflect true feature importance.
  3. Partial Evaluation: Existing metrics often only evaluate the predicted class or ground-truth class, ignoring the rest of the output distribution, which fails to capture the full model behavior.

2. Methodology: Surrogate Faithfulness (SURF)

The authors propose SURF (Surrogate Faithfulness), a new framework designed to measure faithfulness without compromising interpretability.

A. The SURF Surrogate

Instead of complex reconstruction networks, SURF uses a simple, linear surrogate that operates on the model's final linear layer.

• Mechanism: The model's final layer is viewed as a linear projection of embeddings onto class-specific directions. SURF replaces the learned weights with the concept representations ($V$) and concept importances ($A$) derived from the U-CBEM.
• Formula: The surrogate output $\hat{y}_i$ for class $i$ is computed as a linear combination of concept projections weighted by their importances:
  $$\hat{y}_i = \sum_{j} \sum_{k} \alpha_{i,k} P(h_j; V_i)_k$$
  Where $P$ is the projection to concept space, and $\alpha$ represents concept importance.
• Key Advantage: This surrogate has no learnable parameters and is linear. It directly tests if the concepts and their assigned importances can linearly reconstruct the model's output, ensuring the explanation itself is the sole driver of the prediction.

B. The SURF Metrics

To address the limitation of evaluating only the top-predicted class, SURF introduces two metrics that measure error across the entire output distribution:

1. SURFMAE (Mean Absolute Error): Measures the absolute difference between the original model's logits and the surrogate's logits across all classes.
2. SURFEMD (Earth Mover's Distance): Measures the distance between the probability distributions (after softmax) of the original model and the surrogate. This captures how well the surrogate preserves the relative confidence of all classes, not just the winner.

3. Key Contributions

1. Unified Framework: The authors organize existing U-CBEM faithfulness measures into a common framework, identifying that prior methods fail to meet three critical desiderata:
   • Surrogates must be simple (linear).
   • Surrogates must utilize both concept representations and concept importances.
   • Metrics must evaluate errors across all output classes, not just the predicted one.
2. The SURF Metric: They introduce a measure-over-measure study (a sanity check) showing that only SURF correctly degrades in performance as explanations are randomized. Prior methods (like ICE-Eval and C-SHAP-Eval) often failed to detect randomization or reported high faithfulness even with random concepts.
3. Comprehensive Benchmark: They conducted the first large-scale benchmark of U-CBEMs (including CDISCO, ICE, CRAFT, C-SHAP, MCD, HU-MCD, and SAE) across three tasks: Object Classification, Multi-Attribute Prediction, and Age Regression.
4. Hyperparameter Optimization: They demonstrate how SURF can be used to intelligently select the number of concepts ($K$) by observing the trade-off between faithfulness and parsimony (simplicity), rather than setting $K$ arbitrarily.

4. Results

The application of SURF revealed that State-of-the-Art (SOTA) U-CBEMs are largely unfaithful to the models they explain.

• Sanity Check Failure: In "measure-over-measure" experiments where concepts were randomized:
  • SURF correctly showed a significant drop in faithfulness.
  • ICE-Eval reported perfect faithfulness even with random importances (because its surrogate ignored importances).
  • C-SHAP-Eval sometimes reported improved faithfulness with random concepts, indicating a fundamental flaw in its metric.
• Benchmark Findings:
  • Across Object Classification (ResNet-50), Multi-Attribute (MobileNetV2), and Age Regression (ViT), most U-CBEMs produced large errors in both logit space (SURFMAE) and probability space (SURFEMD).
  • SAE (Sparse Autoencoders) performed best among the tested methods but still showed significant errors in probability space.
  • MCD and HU-MCD (which use concept subspaces rather than single vectors) showed better faithfulness than methods using single CAVs, suggesting that representing concepts as subspaces captures more model behavior.
• Parsimony vs. Faithfulness: Increasing the number of concepts did not linearly improve faithfulness for all methods. Some methods (like CDISCO) plateaued or degraded, while MCD/HU-MCD showed a clear plateau, offering a data-driven way to choose the optimal number of concepts.

5. Significance

• Correcting the Field: The paper challenges the prevailing assumption that current unsupervised concept methods are faithful. It exposes that high "faithfulness" scores in previous literature were artifacts of flawed evaluation metrics (complex surrogates and deletion proxies).
• Standardization: SURF provides a rigorous, principled, and computationally cheap standard for evaluating concept-based explanations. It forces researchers to account for concept importances and the full output distribution.
• Practical Impact: By revealing that many visually compelling explanations do not actually reflect the model's computation, the paper urges the community to prioritize faithfulness alongside interpretability. It suggests that future work must adopt SURF to validate claims about concept-based explanations, particularly as these methods are applied to high-stakes domains like healthcare and finance.

In conclusion, the authors argue that without a simple, linear surrogate and comprehensive error metrics, we cannot trust that concept-based explanations are truly faithful. SURF fills this gap, revealing that the current generation of U-CBEMs often fails to faithfully represent the underlying model logic.