Empirical Characterization of Rationale Stability Under Controlled Perturbations for Explainable Pattern Recognition

1. Problem Statement

Current Explainable AI (XAI) evaluation methods are predominantly instance-centric, focusing on generating explanations for individual predictions. They fail to explicitly quantify whether attribution patterns remain consistent across:

• Samples belonging to the same class.
• Small variations (perturbations) of the same input (e.g., paraphrased sentences).

This lack of consistency verification poses a significant risk: a model may appear interpretable in isolation but exhibit unstable reasoning or biased reliance on specific features when faced with similar inputs. This undermines trust, particularly in high-stakes domains like healthcare and finance, where decision-making logic must be reliable and aligned with human expectations.

2. Methodology

The authors propose a framework to measure Rationale Stability using a novel metric called the Explanation Stability Score (ESS).

A. Core Concept: Explanation Stability Score (ESS)

The ESS quantifies the consistency of model explanations by measuring the similarity of feature attribution vectors for inputs that share the same ground-truth label.

• Mechanism: The method computes the Cosine Similarity between SHAP (SHapley Additive exPlanations) value vectors of pairs of inputs $(x_1, x_2)$ where $y(x_1) = y(x_2)$.
• Formula:
  $$ESS = \frac{1}{N} \sum_{i=1}^{N} \text{sim}_{\text{cos}}(S(x_1^{(i)}), S(x_2^{(i)}))$$
  Where $S(x)$ represents the SHAP value vector for input $x$, and $N$ is the number of same-label pairs.
• Interpretation:
  • High ESS (close to 1): The model provides consistent reasoning for similar inputs.
  • Low ESS (close to 0): The model's reasoning is unstable or inconsistent for similar cases.

B. Experimental Setup

• Models: Pre-trained transformer models including BERT, RoBERTa, and DistilBERT.
• Datasets: SST-2 (Stanford Sentiment Treebank) and IMDb (movie reviews) for binary sentiment classification.
• Explanation Tool: KernelSHAP is used to compute feature importance (token-level attributions).
• Perturbation Strategy (Ablation Study): To test robustness, inputs were paraphrased using WordNet synonyms with a 20% replacement probability while preserving the original label.
• Baseline Comparison: The ESS is compared against standard Fidelity Metrics (FID-k), which measure how much the model's confidence drops when the top-$k$ (k=5) most important features are masked.

3. Key Contributions

1. Novel Metric (ESS): Introduction of a quantitative metric to assess explanation consistency across same-label inputs, moving beyond single-instance interpretability.
2. Comprehensive Evaluation: A rigorous empirical study comparing ESS and Fidelity across three transformer architectures (BERT, RoBERTa, DistilBERT) and two datasets, including ablation studies on paraphrased inputs.
3. Detection of Misalignment: Demonstration that ESS can effectively identify "misaligned predictions" where models rely on inconsistent reasoning patterns, even when prediction accuracy remains high.

4. Experimental Results

The experiments revealed distinct behaviors across models and conditions:

• Impact of Perturbations: Paraphrasing inputs consistently reduced the ESS across all models, indicating that explanations are sensitive to minor input variations.
  • Example: For BERT on SST-2 (test set), ESS dropped from 0.1097 (original) to 0.0945 (paraphrased). A t-test confirmed this difference was statistically significant ($p = 0.021$).
• Model Performance:
  • RoBERTa: Exhibited the highest ESS (e.g., 0.1995 on SST-2 test), suggesting the most stable attribution patterns among the tested models. However, it showed very low Fidelity scores ($\approx 0.002$), suggesting potential numerical instability or a disconnect between feature importance and output confidence.
  • BERT: Showed a balanced performance with moderate ESS (0.09–0.18) and higher Fidelity scores compared to RoBERTa. It demonstrated a good trade-off between consistency and capturing feature dependencies.
  • DistilBERT: Displayed the lowest ESS (e.g., 0.0061 on SST-2 test). The authors attribute this to its compact architecture, which may lead to simpler, less variable feature dependencies, but potentially at the cost of nuanced interpretability.
• Class-Specific Instability: Analysis showed that positive sentiment predictions often had lower ESS than negative ones, indicating that the model's reasoning for positive reviews is less stable and more prone to inconsistency.

5. Significance and Conclusion

This work addresses a critical gap in XAI by shifting the focus from "Is this explanation correct for this instance?" to "Is this explanation consistent for similar instances?"

• Trust and Reliability: The study establishes that explanation stability is a prerequisite for human trust. If a model provides contradictory reasoning for semantically similar inputs, human users cannot rely on its decision-making process.
• Beyond Fidelity: The results suggest that standard fidelity metrics alone are insufficient. A model can have high fidelity (masking important features changes the output) but low stability (reasoning changes unpredictably for similar inputs).
• Future Directions: The authors propose integrating the ESS metric directly into model training pipelines to enforce consistency constraints, thereby building AI systems that are not only interpretable but also robustly aligned with human logic across diverse scenarios.

In summary, the paper provides a robust framework for verifying rationale stability, offering a necessary tool for evaluating the reliability of AI systems in practical pattern recognition pipelines.