In defence of post-hoc explanations in medical AI

Technical Summary: In Defence of Post-hoc Explanations in Medical AI

Problem Statement
The paper addresses the "black box problem" in medical Artificial Intelligence (AI), a dual challenge comprising technical opacity and ethical risk. Technically, advanced systems like deep neural networks exhibit training opacity (inability to trace how parameters were induced from data) and inference opacity (inability to discern why a specific input yields a specific output due to model complexity exceeding human cognitive limits). Ethically, this opacity creates a justification problem (clinicians cannot rationally accept outputs they do not understand), an anti-discrimination problem (inability to detect algorithmic bias or spurious reasoning), and a reconciliation problem (inability to resolve disagreements between clinicians and AI).

While Explainable AI (XAI) has promoted post-hoc explanations (models that approximate a black box's reasoning based on input/output records) as a solution, recent critiques argue these explanations are "fool's gold." Critics contend that because post-hoc explanations are merely approximations rather than faithful replications of the underlying reasoning, they fail to improve user understanding, do not reduce safety risks, and may even exacerbate issues like automation bias and overreliance.

Methodology and Theoretical Framework
This article employs a conceptual and argumentative methodology rather than presenting new empirical data. It synthesizes existing literature on XAI, medical ethics, and human-AI interaction to construct a defense of post-hoc explanations. The authors:

1. Define the landscape: Distinguish between model-specific (e.g., saliency maps) and model-agnostic (e.g., LIME) approaches, as well as global and local explanations.
2. Analyze critiques: Systematically review arguments against post-hoc explanations, focusing on their lack of "high-fidelity" (faithfulness to the original model) and empirical findings suggesting they do not improve trust or performance.
3. Construct a counter-argument: Reframe the utility of explanations by drawing parallels to human expert reasoning and scientific modeling, arguing that explanations need not be literal, mechanistic replicas to be functionally useful.
4. Propose mitigation strategies: Identify technical and non-technical methods to mitigate the risks associated with post-hoc explanations (e.g., cognitive forcing, aggregating multiple explanations).

Key Contributions and Arguments
The paper's primary contribution is a nuanced defense of post-hoc explanations against the "high-fidelity" critique, arguing that their value lies in functional understanding rather than mechanistic replication.

• Functional Utility over Fidelity: The authors argue that post-hoc explanations improve the functional understanding of black box systems. Citing Páez (2019), they posit that, like scientific models or thought experiments, explanations can be "literally false" representations of the underlying mechanism yet still provide genuine understanding.
• Improving Clinician-AI Performance: Contrary to claims that explanations are useless, the paper cites empirical evidence (Senoner et al., 2024) showing that radiologist-AI teams using saliency masks achieved 4.7 percentage points higher accuracy in identifying lung lesions. Crucially, explanations helped clinicians distinguish between correct and incorrect AI predictions, leading to better adherence to accurate AI advice and more frequent overruling of inaccurate advice.
• Predictive Capability: The paper highlights that post-hoc explanations (e.g., Wang and Yin, 2021) enhance a user's ability to predict how a system's output will change in response to input variations. This predictive capacity constitutes a form of causal understanding essential for clinical decision-making.
• Justification and Institutional Context: The authors argue that while post-hoc explanations alone may not justify an AI's output, they serve as evidence that, when combined with institutional explanations (details on design decisions, bias mitigation efforts, and developer trustworthiness), allow clinicians to justify their AI-informed decisions.
• Risk Mitigation: The paper acknowledges risks such as automation bias and spurious correlations but argues these are not unique to AI (human explanations also suffer from bias). It proposes specific mitigation strategies:
  • Technical: Using robust local training methods (e.g., "Just Train Twice") and aggregating multiple explanations to reduce misleading signals.
  • Design: Implementing "cognitive forcing" (e.g., delaying recommendations) and "frictional AI" to disrupt heuristic reasoning and encourage analytical thinking.

Results and Claims
The paper does not present new experimental results but synthesizes existing findings to support its thesis. The "results" of the analysis are the following conclusions:

1. Post-hoc explanations do not need to replicate the exact reasoning processes of black box systems to be beneficial.
2. They can improve the accuracy of clinician-AI teams and assist in justifying decisions, provided they are used in conjunction with other strategies.
3. The "black box problem" is not a singular issue solvable by a "silver bullet" (a single technical solution). It is a "wicked" problem requiring a multifaceted approach involving technical tools (like post-hoc explanations and mechanistic interpretability) and non-technical strategies (training, oversight, and post-market surveillance).

Significance
The paper claims significance in correcting the "profound degree of pessimism" currently prevailing in the field regarding post-hoc explanations. By rejecting the "silver bullet" thesis, the authors advocate for a balanced perspective: while post-hoc explanations cannot "solve" the black box problem in one fell swoop, they remain a useful and necessary strategy for mitigating risks in medical AI. The paper emphasizes that dismissing these tools due to their lack of perfect fidelity overlooks their proven capacity to enhance functional understanding and team performance. Ultimately, the solution to the black box problem is framed as a socio-technical endeavor requiring skilled users and effective oversight alongside technological advancements.