💻 computer science

Comparing Four Explanation Methods for Melanoma Classification: Fidelity, Utility and Trust

This study evaluates four post-hoc explanation methods for melanoma classification across fidelity, utility, and trust axes, revealing significant disagreements in their rankings and concluding that multi-axis evaluation with a faithfulness gate is essential for selecting the most appropriate explanation technique.

Original authors: Okoi Michael Obeten, Christiana Uchenna Ezeanya

Published 2026-10-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Okoi Michael Obeten, Christiana Uchenna Ezeanya

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the high-stakes world of skin cancer detection, artificial intelligence has reached a point where computer programs can often spot melanoma as accurately as a human dermatologist. These programs, known as deep learning models, analyze thousands of dermoscopic images—close-up photos of skin lesions—to make a diagnosis. However, these systems operate as black boxes. They provide an answer, such as "this is cancer," but they offer no explanation for why they reached that conclusion. For doctors to trust and use these tools, they need to see the reasoning behind the decision. To bridge this gap, researchers use "explanation methods." These are techniques that generate a visual heat map over the original image, highlighting the specific areas the computer thinks are most important. If the heat map glows brightly over the suspicious mole and fades elsewhere, the explanation seems plausible. But a map that looks right to the human eye is not necessarily a map that reflects how the computer actually thinks.

A team of researchers set out to test whether the visual appeal of these heat maps truly matches their technical reliability and their usefulness to a doctor. They focused on four popular methods used to generate these explanations: Grad-CAM, Integrated Gradients, LIME, and occlusion sensitivity. To ensure a fair test, they trained a single, high-performing computer model to distinguish between benign skin spots and melanoma using a large public dataset of over ten thousand images. Once the model was ready, they applied all four explanation methods to the same set of test images. The researchers then evaluated each method not just by how pretty the map looked, but by three distinct criteria: technical fidelity (does the map actually show what the model is using?), clinical utility (does the map point to the right spot on the skin?), and trust calibration (does the map look better when the model is right than when it is wrong?).

The results revealed a surprising disconnect. The researchers found that the four methods did not agree with each other, and a method that scored highly on one measure often performed poorly on another. For instance, two methods, LIME and occlusion sensitivity, were excellent at pinpointing the exact location of the lesion on the skin. However, a third method, Integrated Gradients, which produced maps that looked quite good on standard tests, failed a crucial reality check. When the researchers scrambled the internal weights of the computer model—essentially turning it into a random guesser—the maps produced by Integrated Gradients remained almost identical to the original ones. This proved that the method was not actually reflecting the model's learned knowledge, but rather reacting to features of the image itself, such as the edges of the lesion. In contrast, the other three methods changed their maps immediately when the model was scrambled, proving they were truly listening to the computer's logic.

When the team looked at how helpful these maps would be to a doctor, the rankings shifted again. LIME placed the most attention directly onto the lesion, but it was extremely slow to generate, taking over six seconds for a single image. Occlusion sensitivity was nearly as accurate at finding the lesion but ran almost thirty times faster, making it a much more practical choice for a busy clinic. Perhaps most importantly, the study showed that a heat map that looks confident does not mean the diagnosis is correct. The researchers discovered that when the computer made a mistake, it often did so with high confidence, producing a very sharp, convincing-looking heat map. This means that a doctor could be misled by a beautiful map that signals certainty for a wrong answer. The study concludes that relying on a single visual check is dangerous. Instead, medical AI systems should be vetted using multiple tests that check for truthfulness, accuracy, and speed, ensuring that the explanations we trust are actually telling the truth about how the machine sees the world.

Technical Summary: Comparing Four Explanation Methods for Melanoma Classification

Problem Statement
Post-hoc explanation methods, such as saliency maps, are widely used to interpret deep learning classifiers for skin-lesion diagnosis. However, these methods are frequently evaluated on a single criterion, typically visual plausibility (whether the heatmap "looks" like it highlights the lesion). The authors argue that this approach is insufficient because visual plausibility does not guarantee that the explanation reflects the model's actual decision-making process (faithfulness), nor does it ensure clinical utility or appropriate trust calibration. There is a lack of comparative studies that evaluate explanation methods simultaneously across three distinct axes: technical fidelity, clinical utility, and trust calibration, particularly within a single reproducible framework.

Methodology
The study employs a unified evaluation pipeline using a binary melanoma classifier trained on the HAM10000 dataset (10,015 dermoscopic images).

Model & Data: An EfficientNet-B0 architecture was fine-tuned on a lesion-level split (7,470 distinct lesions) to prevent data leakage. The model achieved a test AUC-ROC of 0.9295.
Explanation Methods: Four representative post-hoc methods were applied to a stratified subset of 800 test images:
1. Grad-CAM: A gradient-based method.
2. Integrated Gradients (IG): A gradient-based method using a mean-image baseline.
3. LIME: A perturbation-based method using superpixels.
4. Occlusion Sensitivity: A perturbation-based method using sliding windows.
Evaluation Axes:
1. Technical Fidelity: Measured via Intersection-over-Union (IoU) under multiple binarisation rules (fixed thresholds and mass-based), Pointing Game accuracy, Deletion/Insertion Area Under the Curve (DAUC/IAUC), and a cascading parameter-randomisation sanity check (Adebayo et al.).
2. Clinical Utility Proxy: Measured by lesion hit rate (mass inside the lesion), centroid distance (distance from lesion center), and artefact attribution (focus on hair, ink, or rulers).
3. Trust Calibration: Measured by the "fidelity gap" between correctly and incorrectly classified images. The authors controlled for confounds by stratifying analysis by predicted class (True Positives vs. False Positives) and checking correlations with model confidence scores.

Key Results
The study found that the three evaluation axes produced significantly different rankings of the methods, often leading to contradictory conclusions if only one axis were considered.

Fidelity & Sanity Check:
- LIME and Occlusion performed best on localisation metrics (IoU and Pointing Game) and were statistically indistinguishable on IoU ( $p=0.465$ ).
- Integrated Gradients ranked second on Pointing Game and centroid distance but failed the sanity check. It retained a Spearman rank correlation of ~0.33 with its original maps even after seven of nine randomisation stages, indicating it relies heavily on input structure (e.g., edges) rather than the trained model weights.
- Grad-CAM performed poorly on localisation metrics but passed the sanity check.
Utility:
- LIME achieved the highest lesion hit rate (0.4465), placing the most explanation mass inside the lesion.
- Occlusion had the lowest centroid distance (closest to the lesion center) but a lower hit rate, suggesting a diffuse but well-aimed map.
- Grad-CAM had the lowest hit rate and highest centroid distance, attributed to its coarse 7×7 feature grid.
Trust Calibration:
- Unstratified trust scores were negative for all methods, suggesting explanations were "more faithful" on incorrect predictions. This was identified as a confound: errors were predominantly false positives (high-confidence predictions), and fidelity metrics correlated strongly with model confidence ( $\rho > 0.90$ for IAUC).
- When stratified by predicted class, only Grad-CAM showed a significant positive difference in fidelity between True Positives and False Positives. However, the authors note this may still partly reflect confidence effects.
Agreement & Cost:
- Pairwise agreement between methods was low (max rank correlation 0.22).
- LIME was computationally expensive (6.55s/image), while Grad-CAM was fastest (0.021s/image). Occlusion offered a strong balance of quality and speed (0.24s/image).

Key Contributions

Multi-Axis Protocol: A fully computational protocol evaluating explanations across fidelity, utility, and trust simultaneously, demonstrating that single-axis evaluation is insufficient.
Evidence of Divergence: Proof that axes disagree enough to change method selection; specifically, a method (Integrated Gradients) that performs well on localisation fails a fundamental faithfulness test.
Metric Fragility: Demonstration that fixed binarisation thresholds can reverse rankings (e.g., Integrated Gradients appeared to fail localisation at $\tau=0.5$ but recovered under a mass-based rule).
Trust Confound Identification: Evidence that unstratified trust calibration measures track model confidence rather than correctness unless stratified by predicted class.
Practical Guidance: A weighted recommendation framework considering computational cost alongside performance.

Significance and Claims
The paper claims that the disagreement between evaluation axes is substantial enough to alter which explanation method a practitioner would choose. It argues that multi-axis evaluation with a faithfulness gate should become standard practice before explanations are deployed in clinical settings.

Specifically, the authors conclude:

Integrated Gradients (with a mean-image baseline) is not recommended for this architecture due to its failure in the sanity check, despite good localisation scores.
Occlusion emerges as the most defensible default when balancing quality and computational cost.
LIME is preferable when quality is prioritized over latency.
Grad-CAM is suitable when latency is critical and coarse localisation is acceptable.
Trust Calibration: A heatmap's clarity signals model confidence, not necessarily correctness. Relying on map quality as a proxy for correctness can lead to automation bias, especially when errors are high-confidence false positives.

The study emphasizes that no single method led on all three axes and that current practices of relying on visual plausibility or single metrics risk selecting methods that are either unfaithful to the model or misleading to clinicians.

Technical Summary: Comparing Four Explanation Methods for Melanoma Classification

More like this