Assessing Alignment and Stability of Feature Importance Explanations via Weight of Evidence

Technical Summary: Assessing Alignment and Stability of Feature Importance Explanations via Weight of Evidence

Problem Statement
Feature Importance Methods (FIMs) are standard tools in Explainable AI (XAI) for interpreting black-box model predictions. However, existing evaluation metrics often rely on implicit assumptions about what constitutes a "good" explanation, lacking universally accepted criteria for transparency or interpretability. Furthermore, while FIMs provide attribution scores, these scores alone offer limited insight into the underlying reasoning process or the stability of stochastic explainers (e.g., LIME, SHAP). There is a need for a principled framework to evaluate how well FIMs align with prior knowledge, ground truth, or their own internal consistency, particularly in high-stakes domains where trust and accountability are paramount.

Methodology
The authors propose a novel framework that embeds FIMs within a hypothesis-testing paradigm based on the Weight of Evidence (WoE), a statistical concept from information theory.

1. Hypothesis Formulation:

   • The framework treats feature importance as a testable hypothesis ($h$) regarding which features are relevant to a model's decision.
   • A binary hypothesis is derived from continuous attribution vectors ($\alpha$) using selection strategies: Top-k (TK) selection or Adaptive Selection (AS), where features are selected until they explain a sufficient fraction ($\tau$) of the total attribution mass.
   • The reference hypothesis ($h^*$) can originate from three sources:
     • Domain Knowledge: External expert knowledge (e.g., "sex and class are relevant for Titanic survival").
     • Ground Truth: Known relevant features in synthetic datasets.
     • Internal Consistency: The empirical mean attribution derived from multiple runs of the FIM itself.

2. Weight of Evidence Calculation:

   • WoE quantifies the strength of evidence ($e$) supporting hypothesis $h$ over an alternative hypothesis $h'$ (typically the complement $\bar{h}$).
   • Formally: $woe(h : e) = \log \frac{p(e | h)}{p(e | h')}$.
   • Evidence ($e$): Generated by running a stochastic FIM $N$ times on a fixed instance $x$, producing a set of attribution vectors.
   • Prior Probability ($p(h)$): Assumed to be uniform over all $2^d$ possible binary relevance assignments, adjusted for the partial nature of the hypothesis (only specifying relevant features, not irrelevant ones).
   • Posterior Probability ($p(h|e)$): Estimated empirically as the proportion of runs where the derived feature set matches the reference hypothesis $h^*$.

3. Theoretical Connection to Variability:

   • The paper establishes a theoretical link between attribution variability and WoE.
   • Theorem 1: As the variance of the stochastic FIM's attribution scores approaches zero, the WoE for the reference hypothesis (generated from the mean attribution) diverges to $+\infty$ with probability approaching 1. This proves that WoE serves as a direct quantifier of FIM stability.

Key Contributions

• A Formal Strategy for WoE in XAI: The authors adapt the WoE framework to test hypotheses on FIMs, offering a contrastive, evidence-based lens for evaluation.
• Multi-faceted Evaluation: Depending on the source of the reference hypothesis, the framework can:
  • Test alignment between local explanations and global domain knowledge.
  • Measure faithfulness to the model's intrinsic mechanism and alignment with data-level ground truth.
  • Capture the stability (variability) of stochastic FIMs.
• Theoretical Proof: A proof demonstrating that low FIM variability leads to infinite WoE when the reference hypothesis is derived from the mean attribution, formally linking stability to the metric.

Experimental Results
The framework was evaluated using LIME and SHAP across three settings:

1. Domain Knowledge Alignment (Titanic Dataset):

   • Tested against a global hypothesis that "sex" and "pclass" are the primary drivers of survival.
   • Result: LIME showed mixed results (some instances aligned, others divergent), while SHAP produced deterministic explanations that were either perfectly aligned or completely misaligned ($-\infty$ WoE).
   • Insight: WoE effectively identifies regions in the feature space where local behavior deviates from assumed global explanations.

2. Ground Truth and Faithfulness (Synthetic Data):

   • A synthetic dataset was created with known informative features and redundant features with varying noise levels ($\lambda$).
   • Result: At low noise, LIME aligned more with the model (which relied on redundant features) than the ground truth. As noise increased, the model converged to the ground truth, and WoE values reflected this shift.
   • Insight: WoE can localize disalignment along the Data $\to$ Model $\to$ FIM pipeline, diagnosing unfaithful explanations caused by high local curvature or inappropriate kernel widths.

3. Internal Stability (Real-world Datasets):

   • Evaluated on Diabetes, Heart Disease, Mobile, and Churn datasets without external references.
   • Result: SHAP consistently produced the same hypothesis across runs (high stability, $+\infty$ WoE), whereas LIME showed higher variability, especially on complex datasets like Mobile.
   • Insight: WoE successfully captures the internal consistency of FIMs, with positive scores indicating stable explanations and negative scores indicating instability.

4. Theoretical Validation:

   • Experiments varying the number of background samples (for LIME) and adding controlled Gaussian noise to synthetic attributions confirmed that lower variance leads to higher WoE, validating Theorem 1.

Significance and Claims
The paper claims that this framework offers a complementary tool for assessing FIMs. It does not replace existing metrics but provides a principled, statistical method to:

• Quantify how strongly observed evidence supports specific claims about feature importance.
• Distinguish between alignment with prior knowledge, faithfulness to the model, and internal stability.
• Diagnose failures in the explanation pipeline (e.g., when a model learns spurious correlations or when an explainer is too sensitive to noise).

The authors maintain a modest stance, noting that the framework relies on specific design choices (e.g., uniform priors, exact set matching for hypotheses) that could be relaxed in future work. They emphasize that the approach requires multiple runs of the explainer, introducing a computational cost, but argue that stable estimates can be obtained with a relatively small number of runs.