🤖 machine learning

Quantifying Explainable AI-introduced signal noise on ECG data with Spectral Entropy

This paper proposes using spectral entropy to quantify the signal noise introduced by explainable AI (XAI) techniques, demonstrating its effectiveness in distinguishing model signals from XAI artifacts when classifying arrhythmias in ECG data.

Original authors: David A. Kelly, Nathan Blake

Published 2026-06-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: David A. Kelly, Nathan Blake

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, but silent, robot doctor that can look at a heart rhythm strip (an ECG) and tell you if the heart is beating normally or if there's a problem. This robot is great at spotting the trouble, but it can't explain why it thinks that. It just gives you a "Yes" or "No."

To fix this, we use special tools called Explainable AI (XAI). Think of these tools as translators. They look at the robot's decision and draw a highlighter over the parts of the heart rhythm that the robot was looking at. Ideally, the highlighter should only shine on the exact spot where the heart is misbehaving.

The Problem: The "Static" on the Radio
The authors of this paper noticed a problem. Sometimes, these "translator" tools aren't perfect. They don't just highlight the important spot; they also sprinkle in a bunch of random, meaningless dots and lines all over the chart.

The authors call this "self-noise." It's like trying to listen to a clear radio station, but the translator is also playing a lot of static in the background. You can't tell what is the real signal (the robot's actual reasoning) and what is just the translator's own messiness. If the translator highlights too many irrelevant parts, a human doctor might get confused or waste time checking things that don't matter.

The Solution: Measuring the "Chaos"
The paper proposes a new way to measure how much "static" or "noise" a translator tool adds. They use a math concept called Spectral Entropy.

Here is a simple analogy:

Low Entropy (Good): Imagine a choir where everyone sings the same note perfectly together. It's a clear, focused sound. In the paper's terms, this is a good explanation where the highlighter is focused tightly on the one important part of the heart rhythm.
High Entropy (Bad): Imagine a room full of people all shouting different words at once. It's chaotic and hard to understand. This is a bad explanation where the highlighter is scattered everywhere, making it look like white noise.

The authors used this "chaos meter" to test several different translator tools on heart data.

What They Found
They tested the tools on heart rhythms with a specific problem called PVC (a skipped or extra beat). They knew exactly where the problem was (the "ground truth").

The "Clean" Tools: Some tools, like GRAD-CAM, were very quiet. They didn't add much static. However, they sometimes highlighted the wrong area entirely. They were quiet, but they were pointing at the wrong thing.
The "Noisy" Tools: Other tools, like KERNELSHAP, were incredibly noisy. Their output looked almost like pure static. The authors concluded this tool is probably a bad choice for heart data because it's too messy to be useful.
The "Accurate but Noisy" Tools: Some tools, like GRADIENTSHAP, were a bit noisy (they had some extra static), but they did correctly highlight the main problem area.

The Big Takeaway
The main point of the paper is that being quiet (low noise) doesn't always mean being right, but being too noisy definitely makes things harder to understand.

The authors suggest that before we trust a translator tool to help doctors, we should run this "chaos meter" test on it. It's a quick, cheap way to check if the tool is adding too much "garbage" to the explanation. If a tool adds too much noise, it might be confusing the doctor more than helping them, even if the tool thinks it's doing a good job.

In Summary:

AI can spot heart problems but can't explain them.
Explainable AI tools try to explain them but often add "static" (noise) that confuses the picture.
Spectral Entropy is a new tool to measure how much "static" a translator adds.
The Goal: We want translators that are focused (low noise) so doctors can quickly see what the AI is looking at without getting distracted by random highlights.

Technical Summary: Quantifying Explainable AI-introduced signal noise on ECG data with Spectral Entropy

Problem Statement
In healthcare applications, particularly for wearable ECG monitoring, deep learning models require Explainable AI (XAI) tools to justify decisions and facilitate expert verification. However, existing XAI techniques (e.g., SHAP, LIME, GRAD-CAM) rely on heuristics such as sampling and segmentation, which inevitably introduce "self-noise" into the explanation. This noise manifests as spurious signal highlighting features not actually used by the model. A significant gap exists in the literature: nearly half of published XAI applications do not evaluate the explanations themselves, and current guidelines (e.g., FUTURE-AI) recommend quantitative assessment of XAI sensitivity to noise. The authors identify a lack of systematic methods to distinguish between the genuine "model signal" and the "XAI-introduced noise," noting that while accuracy measures exist, they do not quantify the precision or "clutter" of an explanation.

Methodology
The paper proposes Spectral Entropy as a proxy metric to quantify the amount of self-noise added by XAI tools. The authors treat the XAI output (the attribution map) as a signal.

Theoretical Basis: Drawing on Shannon entropy, the authors posit that a "bad" explanation resembles white noise (uniform random allocation of importance), which corresponds to maximum entropy. Conversely, a precise explanation with concentrated signal should exhibit lower entropy.
Calculation: The method involves computing the Power Spectral Density (PSD) of the explanation using a periodogram (via scipy), normalizing the PSD, and applying the standard Shannon entropy formula ( $PSE = -\sum p_i \log p_i$ ). The authors note that while XAI outputs are not time-series, the spectral entropy calculation remains robust regardless of the assumed sample rate.
Experimental Setup:
- Data: 1,000 ECG recordings from the MIT-BIH dataset, classified into 17 arrhythmia types.
- Model: A 17-layer 1D-CNN (trained by the authors based on Sun et al. [15]) achieving 93.5% accuracy.
- XAI Tools Evaluated: GRAD-CAM, LIME (time-series variant), DEEP-LIFT, REX, and two SHAP variants (GRADIENTSHAP, KERNELSHAP).
- Ground Truth: For the Premature Ventricular Contraction (PVC) class, 20 ECGs were manually annotated by domain experts to establish a "ground truth" explanation region.
- Preprocessing: Negative values (indicating features distracting from the classification) were ignored during entropy calculation to focus solely on positive contributions.

Key Results
The study evaluated the average spectral entropy across all 17 classes and specific case studies (PVC):

Noise Levels: The tools fell into two broad categories: high noise and low noise.
- High Noise: KERNELSHAP, LIME, GRADIENTSHAP, and DEEP-LIFT exhibited high spectral entropy (e.g., KERNELSHAP: 10.21, LIME: 10.20), indicating a signal close to white noise or significant spurious peaks.
- Low Noise: GRAD-CAM (1.07) and REX (1.93) showed significantly lower entropy, suggesting less self-noise.
Accuracy vs. Entropy: The authors observed that low entropy does not guarantee accuracy. For instance, LIME had low entropy but failed to highlight the correct PVC region. Conversely, GRADIENTSHAP had high entropy but correctly identified the relevant peaks, albeit with spurious noise.
Consistency: For most tools, entropy remained relatively constant across different arrhythmia classes, suggesting the metric captures the baseline heuristic noise inherent to the tool rather than class-specific variations. REX was an exception, showing high variance, potentially due to its ability to generate multiple explanations.
Specific Findings: KERNELSHAP was deemed unsuitable for this data type due to its noise profile. GRAD-CAM showed minimal noise but suffered from upsampling artifacts that misaligned the signal with the ground truth.

Significance and Claims
The paper claims that spectral entropy serves as an inexpensive, real-time proxy measure for quantifying the "precision" of an explanation, specifically the degree of spurious information added by the XAI tool itself.

Distinction from Accuracy: The authors explicitly state that spectral entropy measures the concentration of the signal (precision) rather than the correctness of the location (accuracy). A low-entropy explanation can still be in the wrong place, but a high-entropy explanation is likely too diffuse to be useful.
Scalability: The primary contribution is a method to scale the assessment of XAI tools beyond the resource limitations of manual expert annotation. It complements existing metrics (like deletion/insertion curves or fidelity) by focusing on the internal noise characteristics of the explanation generation process.
Limitations: The authors remain modest, acknowledging that entropy cannot determine if a model's criteria align with human experts and that it is not a standalone measure of "truthfulness." It is proposed as a tool to augment, not replace, domain expert assessment.

More like this