Trustworthiness Assessment with Explainable AI: A Comparative Study of LIME, Parzen, and Greedy Explanation Methods on Religious Text and Image Classification

Technical Summary: Trustworthiness Assessment with Explainable AI

Problem Statement
As machine learning models are deployed in sensitive domains such as religious discourse classification, the tension between predictive performance and interpretability has intensified. Modern systems often function as "black boxes," making it difficult to verify if decisions rely on genuine semantic understanding or spurious correlations (e.g., author names or header artifacts). While Explainable Artificial Intelligence (XAI) aims to reveal model reasoning, there is a need for rigorous, controlled protocols to assess which explanation methods reliably identify predictions that depend on untrustworthy features. This study addresses the gap in validating the original trustworthiness evaluation of LIME (Local Interpretable Model-agnostic Explanations) on the specific context of religious text and extends the analysis to image classification.

Methodology
The study replicates and extends the experimental framework established by Ribeiro et al. (2016) using the "Religion" subset of the 20 Newsgroups dataset.

• Dataset & Preprocessing: The binary classification task involves distinguishing between alt.atheism and soc.religion.christian (1,310 training, 328 test instances). The text was vectorized into a high-dimensional (98,022 dimensions) binary bag-of-words (BOW) representation. Preprocessing involved removing structural artifacts (headers, routing lines) to minimize metadata leakage, though lexical case was retained to match legacy configurations.
• Classifiers: Four models were trained: Logistic Regression with L2 regularization (LR), Support Vector Machine with RBF kernel (SVM), Random Forest (RF), and K-Nearest Neighbors (K-NN).
• Explanation Methods: Three model-agnostic explanation strategies were evaluated against a Random baseline:
  1. LIME: Perturbs input by removing words, fitting a sparse linear model to the neighborhood.
  2. Parzen Windows: Approximates the black-box globally using kernel density estimation and uses gradient attribution.
  3. Greedy: Iteratively removes the most impactful feature until the prediction changes (counterfactual approach).
• Data-Trusting Protocol: A controlled experiment was conducted where 25% of vocabulary features were randomly designated as "untrustworthy." A prediction was flagged as "untrustworthy" if zeroing these features altered the model's output. The performance of each explainer was measured by its ability to identify these untrustworthy features in its top-K explanations, using Precision, Recall, and F1 scores averaged over five independent rounds with larger evaluation sets (200 instances) to reduce variance.
• Image Extension: A focused case study applied LIME to image classification using InceptionV3 and ResNet18, analyzing the impact of sample size (num_samples) and feature count (num_features) on explanation stability using superpixel perturbations.

Key Results

• Classification Performance: The study successfully replicated the original accuracy benchmarks. Logistic Regression achieved the highest accuracy (95.73%) and F1 score (95.81%), followed by SVM RBF (94.82%), Random Forest (85.67%), and K-NN (74.70%). SVM training was significantly slower (27.86s) compared to LR (0.60s) due to the computational cost of the RBF kernel in high-dimensional space.
• Trustworthiness Identification:
  • LIME: Demonstrated superior performance, achieving near-perfect F1 scores (0.993–1.000) for RF, LR, and SVM. It effectively captured correlated cues within the perturbed neighborhood.
  • Parzen: Showed competitive performance (F1 0.958–1.000) but slightly trailed LIME on LR and SVM.
  • Greedy: Exhibited a paradoxical failure mode. While it achieved perfect precision (identifying only truly important features), its recall was catastrophically low (0.06–0.07) for LR and SVM, resulting in poor F1 scores (0.112–0.124). This is attributed to its minimal nature; it stops after finding one decisive feature, missing the collective weight of multiple weak, untrustworthy features common in sparse text spaces.
  • Random: Showed high precision but near-zero recall, an artifact of high model confidence where few instances actually flipped, making random guessing appear precise but useless.
• Image Stability: In the image experiments, explanation stability was found to be highly dependent on sample size. Explanations with 10 samples were unstable and scattered, whereas 1,000 samples produced consistent, semantically focused regions. A trade-off was observed between runtime and stability, with 1,000 samples taking approximately 68× longer than 10 samples.

Significance and Claims
The paper claims to provide a faithful replication of the original LIME trustworthiness evaluation, confirming that LIME is a strong practical option for identifying untrustworthy features in sparse, high-dimensional text classification tasks involving religious discourse. The study offers a technical explanation for the "failure" of the Greedy approach in this specific context, highlighting that minimal contrastive explanations may lack recall in settings where untrustworthy features are individually weak but collectively decisive.

The authors maintain a modest scope, noting that the findings are bounded by the binary nature of the dataset, the use of sparse BOW features rather than contextual embeddings, and the limited scope of the image analysis. The study does not claim universal superiority for LIME across all domains or model families but validates its efficacy within the specific experimental protocol and dataset constraints used. The work serves as a technical extension of the original 2016 study, adding a fourth classifier (K-NN), analyzing stability under repeated runs, and providing a preliminary look at LIME's behavior in deep image classification.