Challenges in Explaining Pretrained Clinical Text Classifiers

Technical Summary: Challenges in Explaining Pretrained Clinical Text Classifiers

Problem Statement
The integration of deep learning models into high-stakes healthcare domains, particularly for analyzing unstructured Electronic Health Records (EHRs), is hindered by a lack of transparency. While models achieve strong predictive performance, their "black box" nature limits trust and accountability. Existing Explainable AI (XAI) techniques, such as LIME (Local Interpretable Model-Agnostic Explanations) and SHAP, were primarily designed for tabular or image data. When applied to long, noisy, and heterogeneous clinical narratives, these post-hoc methods often fail to provide clinically meaningful insights. The paper identifies a critical gap: current token-level and perturbation-based explanation strategies frequently highlight trivial tokens, ignore multi-word clinical concepts, and produce unstable or misleading attributions that do not align with expert reasoning.

Methodology
The authors conducted a targeted empirical evaluation using a real-world dataset of 467 de-identified hospital discharge summaries from the MIMIC-IV database. The dataset consists of adult patients diagnosed with kidney stones (ICD-10 code N20.0), labeled for a binary classification task: predicting whether the hospital length of stay (LOS) exceeded the cohort median.

1. Model Training: A domain-adapted ModernBERT model was fine-tuned on the dataset using the HuggingFace Transformers framework. Inputs were truncated to 1024 tokens, and the model was optimized using cross-entropy loss.
2. Explanation Generation: The authors employed LIME to generate token-level importance scores for the model's predictions.
3. Three-Part Demonstration: To expose the limitations of LIME in this context, the authors performed three specific analyses:
   • Sensitivity Analysis: A deletion-based experiment where the top-ranked tokens identified by LIME were progressively removed to observe the impact on model confidence.
   • Token Frequency Analysis: An aggregation of the top-10 attributed tokens across 20 diverse discharge summaries to identify the prevalence of non-informative tokens (e.g., stopwords, symbols).
   • Off-Manifold Perturbation: A synthetic experiment where a medically plausible note was perturbed by randomly masking tokens to create syntactically broken or semantically incoherent inputs, testing whether the model and explainer would still yield high-confidence, interpretable results.

Key Contributions and Results
The study empirically demonstrates six core challenges inherent to applying standard post-hoc explainers to clinical text:

1. Disproportionate Influence of Top Tokens: Sensitivity analysis revealed that model confidence drops sharply only when the top one or two tokens are removed. Subsequent removals of lower-ranked tokens (even those with non-negligible LIME scores) had minimal impact. This suggests that long lists of "important" tokens provided by explainers often lead to the over-interpretation of marginal features.
2. Prevalence of Non-Informative Tokens: Frequency analysis showed that LIME frequently assigns high attribution scores to stopwords, formatting artifacts (e.g., underscores), and function words (e.g., "no," "and") rather than clinically significant terms. While filtering can remove some noise, it risks discarding context-dependent medical phrases.
3. Failure to Capture Multi-Word Concepts: Token-level attribution fails to capture medical concepts that span multiple words (e.g., "chronic kidney disease"), instead isolating individual words that lack semantic meaning in isolation.
4. Vulnerability to Off-Manifold Perturbations: The perturbation experiment revealed that the model maintained high confidence (exceeding 0.75) for 17.2% of syntactically broken and semantically incoherent inputs. This indicates that perturbation-based methods can generate explanations for inputs that lie far outside the training data manifold, distorting the explanation process.
5. Dilution and Instability: The reliance on random sampling in LIME leads to instability, where repeated runs yield different attributions. Furthermore, averaging effects across many perturbed samples can obscure localized patterns, producing diffuse explanations.

Significance and Claims
The paper argues that the limitations observed are not isolated artifacts but stem from structural flaws in token-level and perturbation-based explainers when applied to complex clinical narratives. The authors claim that for XAI to be viable in healthcare, explanation strategies must move beyond isolated tokens to incorporate:

• Linguistic Coherence: Methods that respect multi-word expressions and medical entities.
• Semantic Grounding: Explanations that align with clinical reasoning rather than statistical token frequency.
• Robustness: Strategies that are stable across runs and resistant to linguistic noise.

The authors conclude that building clinically useful interpretability tools requires not only technical advancements in handling longer textual units but also human-centered evaluation involving medical experts to ensure explanations align with decision-support needs. They emphasize that these challenges extend beyond medicine to other high-stakes textual domains, such as legal and scientific NLP.