When Explanations Mislead: A Systematic Audit of Saliency Map Localization Failures in Chest X-Ray AI Despite Correct Predictions

Technical Summary: When Explanations Mislead

Problem Statement
The paper addresses a critical safety gap in clinical radiology AI: the uncritical assumption that correct predictions by deep learning models are inherently accompanied by accurate spatial explanations. Saliency maps (e.g., GradCAM, Integrated Gradients) are widely used as the de facto explanation layer to build clinician trust. However, the implicit belief that a model identifying a pathology (e.g., consolidation) is necessarily attending to the correct anatomical region remains unverified at scale. The authors identify a specific "dangerous failure" mode: cases where a model makes a correct prediction with high confidence, yet the generated saliency map fails to localize the pathology, potentially misleading clinicians who rely on the visual explanation to validate the AI's output.

Methodology
The study conducts a systematic audit using the VinDr-CXR dataset, which features multi-radiologist consensus bounding boxes for 28 pathological findings.

• Dataset & Sampling: From 18,000 images, the authors selected a stratified sample of 416 images across six pathologies (Atelectasis, Cardiomegaly, Consolidation, Nodule/Mass, Pleural Effusion, Pneumothorax). Ground truth was defined as the mean of three radiologist annotations.
• Model: A pre-trained DenseNet-121 (from TorchXRayVision) was used as the baseline. The model was not fine-tuned on VinDr-CXR to prevent data leakage. Predictions were considered positive if the confidence exceeded 0.5.
• XAI Methods: Four post-hoc explainability methods were applied to the 336 images that received correct positive predictions:
  1. GradCAM and GradCAM++: Gradient-based class activation maps.
  2. Integrated Gradients: Axiomatic attribution along a path from a baseline.
  3. LIME: Local surrogate modeling via superpixel perturbations.
• Evaluation Metrics:
  • Intersection over Union (IoU): Measured overlap between the binarized saliency map (top 50% activation) and the radiologist ground truth.
  • Pointing Game Accuracy: A binary metric scoring 1 if the pixel of maximum activation fell within the ground truth box.
  • Dangerous Failure Rate: The proportion of correct predictions where the IoU fell below 0.1. This threshold was selected to represent a clinically meaningless overlap that could actively misdirect attention.

Key Results
The audit reveals a severe decoupling between prediction accuracy and explanation fidelity:

• High Failure Rate: Across all four XAI methods, 83.0% of correctly classified images constituted "dangerous failures" (IoU < 0.1).
• Low Localization Performance:
  • Mean IoU: Ranged from 0.044 (LIME) to 0.062 (Integrated Gradients).
  • Pointing Game Accuracy: Ranged from 1.2% (LIME) to 27.4% (Integrated Gradients).
• Method Comparison: Integrated Gradients performed best but still failed in 76.2% of cases. LIME performed worst, with a 92.0% failure rate.
• Pathology Variation: Conditions occupying large anatomical regions (Cardiomegaly, Pleural Effusion) showed slightly higher IoU values, while focal conditions (Nodule/Mass, Pneumothorax) showed the lowest localization accuracy.
• Qualitative Evidence: Representative cases (e.g., Atelectasis) showed models correctly predicting the condition with high confidence (e.g., 63%) while the saliency map concentrated on irrelevant regions (e.g., lung bases/diaphragm) rather than the annotated pathology.

Key Contributions

1. Quantification of Decoupling: The study provides empirical evidence that in current chest X-ray AI (specifically DenseNet-121), high prediction accuracy does not guarantee spatially faithful explanations.
2. Definition of "Dangerous Failure": The authors formalize a failure mode distinct from simple misclassification: a correct prediction paired with a misleading explanation, which poses a unique risk to clinical safety by exploiting clinician trust.
3. Benchmarking XAI Methods: The paper establishes a comparative baseline for GradCAM, GradCAM++, Integrated Gradients, and LIME on the VinDr-CXR dataset, demonstrating that even the "best" performing method (Integrated Gradients) fails to meet clinical localization standards.

Significance and Claims
The paper argues that the current practice of deploying XAI-augmented systems based solely on prediction accuracy is fundamentally flawed. The authors claim that:

• Visual Plausibility is Insufficient: Heatmaps that appear anatomically plausible can be systematically wrong in their localization.
• Mandatory Validation Gates: Prediction accuracy alone is insufficient for clinical deployment. The authors recommend that localization validation against radiologist ground truth become a mandatory step before XAI tools are integrated into radiological workflows.
• Specific Recommendations:
  • A minimum pointing game accuracy of 60% should be established as a deployment threshold.
  • Integrated Gradients is preferred over GradCAM variants and LIME for chest X-ray tasks, though none currently meet the proposed safety standards.
  • LIME should not be used for localization in this setting due to its near-zero pointing accuracy.

The study concludes that the assumption "correct prediction implies trustworthy explanation" is not supported by data, and that without rigorous localization validation, saliency maps may actively degrade clinical decision-making rather than support it.