💻 computer science

When explanations disagree: a consequence-calibrated audit of post-hoc attribution for clinical time-series prediction

This study demonstrates that post-hoc attribution methods for clinical time-series predictions often yield materially inconsistent and clinically consequential disagreements, suggesting that consequence-calibrated auditing is more valuable than seeking a universally preferred explanation method.

Original authors: Ahmed Mohammed Fawaz

Published 2026-10-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ahmed Mohammed Fawaz

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In modern hospitals, computers are increasingly helping doctors make life-or-death decisions by analyzing the steady stream of data coming from patients in intensive care units. These systems watch heart rates, blood pressure, and breathing patterns, looking for subtle signs that a patient is about to get worse. Because these decisions carry such heavy weight, regulators and doctors demand to know why the computer made a specific prediction. They want an explanation that points to the exact vital sign or historical event that triggered the alarm. To provide this, researchers have developed tools that act like a spotlight, highlighting the most important parts of a patient's data after the computer has already made its calculation. The prevailing hope has been that no matter which spotlight tool is used, it will shine on the same features, giving a consistent story about why the patient is at risk.

A researcher set out to test whether this hope was reality or an illusion. They gathered data from nearly 29,000 hospital admissions across three massive, independent databases, covering a wide range of critical conditions like brain injuries and spinal surgeries. They trained a computer model to predict whether a patient would survive their hospital stay, and then they applied three different spotlight tools to the same model and the same patients to see if the tools agreed on what mattered most. The results were stark: the tools rarely agreed. On average, the different methods only aligned about half the time on which factors were most important. In no single group of patients did the tools reach a level of agreement that experts consider substantial. The lack of consensus was not due to the specific type of illness the patients had, but rather depended entirely on which database the data came from, suggesting that the explanation is a property of the tool and the data source, not a fixed truth about the patient.

To understand if this disagreement actually mattered, the researcher created a series of simulated patient cases where they knew the exact cause of the outcome in advance. They injected specific signals into the data, such as a sudden drop in blood pressure, and watched to see if the spotlight tools correctly identified these signals. The results showed that no single tool was consistently the best. One tool might perfectly identify a fleeting signal in one scenario but fail completely in another, while a different tool would do the opposite. In fact, the ranking of which tool was best would completely flip depending on the specific condition of the patient. This means that choosing a single "best" tool to standardize across all hospitals would be a mistake, as it would be right for some patients and wrong for others.

The most critical finding emerged when the researcher translated these technical disagreements into potential medical actions. They mapped the highlighted features to specific interventions a doctor might take, such as giving fluids, starting a medication to raise blood pressure, or ordering a scan. When two different tools were applied to the same patient, they pointed to different treatments in about two-thirds of the cases. This suggests that the choice of explanation tool could fundamentally change the advice given to a doctor. The researcher also tested whether a disagreement between tools could serve as a warning sign that the computer's prediction was wrong. Surprisingly, they found the opposite: when the tools disagreed, the computer's prediction was actually less likely to be wrong, not more. This counterintuitive result implies that using disagreement as a simple alarm for error could be misleading.

Ultimately, the study concludes that we cannot simply pick one explanation method and trust it universally. Instead, the behavior of these tools is so dependent on the specific data and the patient's condition that a new approach is needed. The researcher proposes that before any such system is used in a hospital, it must undergo a specific kind of audit that measures not just if the computer is accurate, but how much the explanations vary and what those variations mean for patient care. They found that while they could predict when disagreements were likely to happen based on how closely related the patient's vital signs were to each other, they could not find any specific group of patients where the explanations were reliable enough to be trusted without question. The path forward, they argue, is not to standardize on a single tool, but to constantly check and calibrate these explanations against the real-world consequences of the decisions they might inspire.

Technical Summary: A Consequence-Calibrated Audit of Post-Hoc Attribution for Clinical Time-Series Prediction

Problem Statement
The deployment of clinical prediction models in electronic health records (EHR) increasingly relies on post-hoc attribution methods (e.g., SHAP, Integrated Gradients) to explain model decisions. This practice rests on a critical, untested assumption: that different attribution methods applied to the same model and patient will yield materially consistent accounts of the prediction drivers. If explanations vary significantly by method, the transparency they purport to offer is illusory, and the choice of attribution becomes a clinical decision rather than a technical detail. Current literature lacks large-scale empirical measurement of explanation agreement on clinical time-series data and, crucially, lacks a framework to translate statistical disagreement into statements about patient care consequences.

Methodology
The author conducted a consequence-calibrated audit across 13 ICU cohorts (28,875 admissions) spanning five neurocritical/neurosurgical populations from three independent databases: MIMIC-III, MIMIC-IV, and eICU-CRD.

Modeling and Data: Multilayer perceptrons (MLPs) were trained to predict in-hospital mortality using flattened hourly vital sign windows. Subject-level splits ensured no data leakage. A rigorous pre-registered leakage audit excluded any cohort where features could predict the outcome with perfect accuracy (AUROC=1.0) prior to explanation generation.
Attribution Panel: Three gradient-based methods were applied: Gradient Saliency, Gradient×Input, and Integrated Gradients. Agreement was measured using pairwise Kendall's $\tau$ on feature importance rankings.
Faithfulness Benchmarking: Since real clinical data lacks ground truth for causal drivers, the author constructed seven semi-synthetic cohorts with known injected causal structures (ranging from easy cases to transient signals, interactions, and distractors). Benchmarks were gated by an "admissibility check": a model was only tested if it could discriminate the outcome and lose that discrimination when the injected causal signal was ablated via a counterfactual operator (to avoid residual signal issues with zero-ablation).
Consequence Calibration: To determine if disagreement matters clinically, the author mapped attributions to a structural action space (e.g., fluids, vasopressors, antibiotics). They calculated a "decision-flip rate": the proportion of cases where two methods implicated different interventions.
Predictability and Trust: A model was fitted to predict per-case disagreement using pre-computable covariates (feature correlation, temporal signal concentration, prediction margin). The author also tested if disagreement correlated with model misclassification.
Clinical Validation: A vignette survey of 35 clinicians (Neurosurgery, Neurocritical Care, General Critical Care) assessed whether the structural action categories represented genuine clinical disagreement.

Key Results

Moderate and Source-Dependent Agreement: Across 13 cohorts, the three attribution methods showed only moderate agreement (mean Kendall $\tau$ = 0.50; range 0.41–0.61). No cohort reached the pre-specified threshold for substantial agreement ( $\tau \ge 0.70$ ). Agreement varied systematically by data source (explaining 63% of variance) rather than by clinical population.
No Universally Superior Method: In semi-synthetic benchmarks with known ground truth, no method was consistently most faithful. Rankings were condition-dependent; for instance, Gradient Saliency perfectly recovered transient signals but ranked last for early signals, while Integrated Gradients showed the inverse pattern.
Consequential Disagreement: Disagreement frequently translated into different recommended actions. In semi-synthetic data, two method pairs (Saliency vs. Gradient×Input; Saliency vs. Integrated Gradients) produced different actions in 62–66% of cases. This "far/close" pattern replicated in a real-world cohort (MIMIC-IV spinal fusion), where flip rates were 67–69% for the divergent pairs versus only 6.7% for the Gradient×Input vs. Integrated Gradients pair.
Predictability of Disagreement: A model predicting disagreement achieved an out-of-sample AUC of 0.73. Feature correlation was the dominant predictor (higher correlation reduced agreement), while temporal concentration and prediction margin were not significant. However, no stratum of cases met the pre-registered threshold for "trustworthy" explanations ( $\tau \ge 0.70$ ).
Disagreement Does Not Predict Misclassification: Contrary to the hypothesis that disagreement flags model errors, higher disagreement was associated with lower odds of misclassification (net of prediction confidence) in real data. This suggests disagreement is not a reliable proxy for predictive uncertainty.
Clinical Validity of Action Space: The clinician vignette survey confirmed that the action categories used in the audit correspond to points of genuine, often specialty-patterned clinical disagreement (e.g., neurosurgeons vs. intensivists choosing between vasopressors and fluids), validating the structural mapping's face validity.

Significance and Claims
The paper argues against the standardization of a single "best" attribution method for clinical time-series prediction, as method rankings are unstable and condition-dependent. Instead, it proposes a shift in regulatory and clinical practice:

Consequence-Calibrated Auditing: Rather than selecting a preferred method, stakeholders should audit explanation behavior to characterize when explanations are likely to disagree.
External Validation of Explanation Behavior: Explanation stability is not a property that transfers with a model across different data sources. An explanation audit conducted on one database provides little assurance for another, necessitating external validation of explanation behavior just as is required for predictive performance.
Risk Stratification: While a hard "trust boundary" (a subset of cases where explanations are guaranteed reliable) could not be identified, a continuous disagreement-risk score can be computed at prediction time to flag cases where explanation instability is likely.
Cautious Interpretation of Disagreement: The finding that disagreement does not predict model error (and may inversely correlate with it) cautions against using disagreement scores as a proxy for predictive risk or uncertainty.

The study concludes that post-hoc explanations on clinical time series are not yet ready to bear regulatory weight unconditionally. The evidence supports requiring a consequence-calibrated audit as part of the evidence package for clinical model deployment, acknowledging that explanation behavior is a property of the method-data interaction, not just the model.

More like this