How Best to Explain Machine Learning Models to Clinicians: A User Study of Explanation Types

Problem
Machine learning (ML) models are increasingly deployed in clinical settings to assist with decision-making, yet many high-performance models (e.g., neural networks, gradient-boosted trees) operate as "black boxes," making their internal logic difficult for clinicians to interpret. While Explainable Artificial Intelligence (XAI) methods have been developed to characterize the relationship between model inputs and predictions, it remains unclear which specific types of explanations are most valued by clinicians. There is a gap in understanding how different explanation modalities—specifically attribution, counterfactual, and rule-based explanations—affect critical dimensions of clinical interaction, including trust in the model, understanding of the model's logic, comprehension of the explanation itself, and the influence of the explanation on clinical reasoning regarding patient status.

Methodology
The authors conducted a user study involving 39 clinicians (nurses, physician trainees, and attending physicians) from critical care and hospital medicine departments at the University of Wisconsin-Madison and the University of Chicago. The study utilized a deep learning model based on an LSTM architecture, trained on longitudinal patient data (demographics, labs, vital signs) to predict the risk of clinical deterioration (death or ICU transfer) within 24 hours.

The study design involved presenting participants with five synthetic, high-risk patient cases. For each case, participants were asked to:

1. Review patient feature data and rank the top five features they believed indicated high risk.
2. View explanations of the model's prediction generated by three distinct XAI methods:
   • Attribution: Using Integrated Gradients to assign importance scores to features over time.
   • Counterfactual: Using CoMTE to show how modifying specific feature values would change the prediction.
   • Rule-based: Using Anchors to identify sufficient conditions (IF-THEN rules) for the prediction.
3. Re-rank their feature importance selections after viewing each explanation.
4. Rate their agreement with statements measuring model trust, model understanding, explanation understanding, and the influence of the explanation on their thinking.

The study employed Linear Mixed-Effect Models (LMMs) to analyze the data, treating participant ID as a random effect and controlling for variables such as role, recruitment site, gender, age, and baseline trust. Statistical significance was determined using Z-tests with a 95% confidence threshold.

Key Contributions

1. Comprehensive Evaluation: The paper provides an in-depth evaluation of attribution, counterfactual, and rule-based explanations generated by XAI methods for a real clinical deep learning model, assessing them across four essential dimensions: model trust, model understanding, explanation understanding, and influence.
2. Comparative Analysis: It offers a direct comparison of these three explanation types to identify which specific modalities most effectively impact clinicians' trust, understanding, and decision-making processes.
3. Implementation Guidance: The study derives specific guidance for implementing XAI in clinical settings based on clinician feedback, including preferences for explanation representation and the potential for role-specific utility.

Results

• Explanation Importance: Clinicians generally consider explanations important. Physicians reported a statistically significant increase in the perceived importance of explanations after interacting with them, whereas nurses' perceived importance remained stable.
• Model Trust and Understanding: All three explanation types increased clinicians' trust in the model and their understanding of how the model makes predictions. However, attribution explanations consistently demonstrated the most significant positive effects across all measured dimensions compared to counterfactual and rule-based explanations.
• Explanation Understanding: Participants found attribution explanations the easiest to understand. While all types were generally understood, attribution scores were significantly higher than the other two types in the case-by-case analysis.
• Influence: Attribution explanations were the most influential, causing the greatest number of changes in participants' rankings of important features (measured by Count of Rank Changes) and receiving the highest self-reported influence scores.
• Preferences: Nearly half of the participants (46.2%) expressed a preference for viewing multiple explanation types simultaneously. When selecting combinations, attribution and counterfactual explanations were most frequently paired. Additionally, clinicians strongly preferred concise representations (e.g., top features or short condition lists) over detailed, exhaustive lists.

Significance
The paper claims that providing explanations for ML predictions in clinical settings is crucial for improving the utility of these systems. The findings suggest that when implementing XAI in clinical workflows, developers should prioritize attribution explanations as the primary modality due to their superior performance in building trust, fostering understanding, and influencing clinical reasoning. However, the authors also recommend allowing for the display of multiple explanation types to accommodate diverse clinician preferences and workflows. Furthermore, the study highlights that explanation development should be tailored to specific clinical roles, as nurses and physicians may utilize and value explanations differently to support their respective tasks. The authors conclude that these insights can lead to changes in ML-assisted clinical practice and potentially improve patient outcomes by ensuring clinicians can effectively interpret and act upon model predictions.