🤖 AI

Position: Explanation Stability Is a Property of the Model Method Pair, Not the Model

This position paper argues that explanation stability is not an intrinsic property of a model but an emergent characteristic of the specific model-method pair, necessitating cross-method validation to avoid misleading claims of safety or reliability.

Original authors: Kabilan Elangovan, Daniel Ting

Published 2026-07-21

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Kabilan Elangovan, Daniel Ting

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out why a robot chef decided to burn your toast. You ask the robot, "Why did you do that?" and it points a glowing finger at the bread. But here's the catch: the robot has two different ways of pointing. One way, let's call it "The Laser Pointer," highlights the exact spot where the bread was hottest. The other way, "The Spotlight," shines a broad beam over the whole toaster, emphasizing the general area. If you only use the Laser Pointer, you might think the robot is very consistent. But if you switch to the Spotlight, the robot might suddenly point at a completely different part of the kitchen! This is the heart of a growing problem in the world of Artificial Intelligence (AI), specifically in a field called Explainable AI (XAI).

In the world of AI, we build "models" (think of them as super-smart digital brains) to make decisions, like spotting diseases in X-ray pictures. But these models are often "black boxes"—we know what goes in and what comes out, but we don't always know how they decided. To fix this, scientists use "attribution methods." These are tools that act like highlighters, showing us which parts of an image (like a dark spot on an X-ray) were most important for the AI's decision. For a long time, researchers assumed that if an AI model was stable and reliable, its highlighter would always point to the same spot, no matter which tool you used to do the highlighting. This paper argues that this assumption is wrong. It suggests that the "stability" of an explanation isn't a property of the AI brain itself, but rather a result of the specific highlighter tool you happen to be using. If you change the tool, the story the AI tells might change completely, even if the AI's final answer stays the same.

The Great Highlighter Swap

The authors of this paper, Kabilan Elangovan and Daniel Ting, decided to put this idea to the test with a controlled experiment using chest X-rays. They took three famous AI models—DenseNet201, ResNet50V2, and InceptionV3—and taught them to spot five different lung conditions, including pneumonia and COVID-19. They trained these models until they were incredibly good at their job, achieving a success rate (AUC) of over 99% for all three. In other words, they were all equally brilliant at diagnosing the disease.

Then came the twist. The researchers asked these models to explain why they made their diagnoses using two very different highlighter tools: LayerCAM and Grad-CAM++.

LayerCAM is like a high-definition microscope. It looks at the image pixel by pixel, preserving fine details and local textures.
Grad-CAM++ is more like a wide-angle lens. It looks at the bigger picture, using complex math to weigh different parts of the image based on their overall importance.

The researchers wanted to see: If they tweaked the models slightly (a process called "fine-tuning"), would the highlighters stay steady? Would they keep pointing at the same lung spots?

The Plot Twist: It Depends on the Tool

Here is where the story gets wild. The results showed that the "stability" of the explanation depended entirely on which highlighter you used. It was like asking three different people to describe a painting; one person might say, "This painting is stable because the colors don't change," while another says, "No, the shapes are shifting wildly." Both could be right, but they are using different rules.

The InceptionV3 Surprise: When the researchers used LayerCAM (the microscope), InceptionV3 looked like the most stable model. It kept its focus tight, with a stability score (called IoU) of 0.777. It seemed like the perfect, reliable AI.
The Grad-CAM++ Shock: But when they switched to Grad-CAM++ (the wide-angle lens), InceptionV3 fell apart. Its stability score dropped to 0.643, a massive 17.3% degradation. Suddenly, the model that looked so steady was actually very shaky, just because the tool changed.
The DenseNet201 Hero: On the other hand, DenseNet201 was the calm in the storm. Whether they used the microscope or the wide-angle lens, its stability score barely changed. It went from 0.699 to 0.690, a tiny 1.3% variation. This model seemed to have a "method-agnostic" nature, meaning its explanations stayed consistent regardless of the tool.
The ResNet50V2 Struggle: ResNet50V2 had a rough time. It started with a moderate score of 0.519 under LayerCAM but crashed to 0.383 under Grad-CAM++. It was unstable no matter what, but the drop was especially dramatic.

The most shocking part? All three models were still getting the diagnosis right 99% of the time. Their "answers" were perfect, but their "reasons" were shifting around like sandcastles in the wind, depending on which tool you used to look at them.

Why This Matters: The "Illusory Safety" Trap

The paper argues that this discovery breaks a major rule in how we trust AI. Currently, if a researcher says, "Our AI model is stable and trustworthy," they often prove it by showing results from just one highlighter tool. This paper says that is scientifically invalid. It's like claiming a car is "safe" because it passed a crash test with a specific type of airbag, but never testing it with a different one.

The authors point out that if you only look at InceptionV3 through the LayerCAM lens, you might think it's the best choice for a hospital. But if the hospital actually uses Grad-CAM++ (or if the software updates and switches tools), the AI's explanations could become unreliable, potentially misleading doctors. The paper suggests that explanation stability is not a trait of the model itself, but a property of the "Model + Method" pair. You cannot separate the two.

The Takeaway: Check Your Tools

So, what should we do? The authors propose a few changes to how we study and regulate AI:

Don't trust a single tool: Researchers should never claim a model is "stable" unless they have tested it with at least two different highlighter methods. If the results change wildly, the model isn't stable; it's just lucky with that one tool.
Regulators need to be specific: When medical devices are approved by agencies like the FDA, the specific highlighter tool used to generate the explanation must be locked in. If a hospital changes the tool later, the device might need to be re-approved, because the "reason" the AI gives could change.
Pick the right architecture: If you are building a medical AI, you might want to choose a model like DenseNet201, which seems to hold its ground across different tools, rather than InceptionV3, which is fragile and depends heavily on the specific method used.

In short, this paper is a wake-up call. It tells us that in the world of AI, the "truth" of an explanation isn't just about the AI's brain; it's also about the glasses we use to look at it. If we want to trust AI in life-or-death situations like medicine, we need to make sure our explanations hold up, no matter which pair of glasses we put on.

Technical Summary: Explanation Stability Is a Property of the Model–Method Pair, Not the Model

Problem Statement
The paper argues that current claims regarding "explanation stability" in Explainable AI (XAI) are scientifically invalid when derived from single-method evaluations. Just as statistical significance requires specifying a test statistic, explanation stability must be scoped to the specific attribution method used. The authors posit that stability is not an intrinsic property of a machine learning model but an emergent property of the interaction between the model architecture and the attribution method. Without cross-method validation, researchers risk conflating method-specific behaviors with model behaviors, leading to illusory safety assurances in safety-critical domains like medical imaging.

Methodology
To demonstrate this method-dependency, the authors conducted controlled experiments on chest X-ray classification involving five classes (Normal, Pneumonia, Tuberculosis, COVID-19, Lung Opacity).

Architectures: Three ImageNet-pretrained models were evaluated: DenseNet201, ResNet50V2, and InceptionV3.
Training Protocol: Models underwent a two-phase training process: transfer learning (epochs 1–10, frozen backbone) and fine-tuning (epochs 11–20, unfrozen layers).
Attribution Methods: Two gradient-based attribution methods with distinct mathematical objectives were applied to the penultimate convolutional layers:
- LayerCAM: Utilizes pixel-wise gradients to preserve fine-grained spatial detail.
- Grad-CAM++: Incorporates higher-order gradients with adaptive weighting for multi-instance localization.
Experimental Controls:
- Semantic Drift Analysis: The study compared attribution maps from epoch 8 (transfer learning plateau) and epoch 19 (fine-tuning convergence) to quantify explanation evolution.
- True-Positive Filtering: To isolate explanation evolution from prediction quality changes, analysis was restricted to samples correctly classified by all three architectures in both training phases (2,430 samples).
- Metrics: Explanation changes were quantified using Overlap IoU (primary), Spatial Displacement, Pattern Correlation, and Concentration Change. Results were weighted by inverse class frequency to address severe class imbalance.

Key Results
The experiments revealed that attribution method choice completely reverses the perceived stability rankings of the architectures, despite all models achieving equivalent predictive performance (>99% AUC) after fine-tuning.

Method-Dependent Rankings:
- Under LayerCAM: InceptionV3 demonstrated superior stability (IoU = 0.777), outperforming DenseNet201 (0.699) and ResNet50V2 (0.519).
- Under Grad-CAM++: The ranking reversed. DenseNet201 became the most stable (IoU = 0.690), while InceptionV3 dropped significantly (0.643), and ResNet50V2 exhibited catastrophic degradation (0.383).
Cross-Method Robustness:
- DenseNet201: Showed method-agnostic stability with only a 1.3% variation in IoU between methods, suggesting dense connectivity promotes coherent explanation evolution.
- InceptionV3: Exhibited a 17.3% degradation in stability when switching from LayerCAM to Grad-CAM++, indicating that its multi-scale pathways produce stable local activations but volatile cross-scale interactions.
- ResNet50V2: Demonstrated universal instability across both methods, with a 26.2% cross-method degradation, suggesting residual shortcuts enable dramatic reorganization without structural coherence.
Statistical Significance: All architecture comparisons achieved $p < 0.001$ . InceptionV3's cross-method difference showed a large effect size (Cohen's $d = 1.47$ ).

Key Contributions

Empirical Demonstration of Method-Dependency: The paper provides controlled evidence that explanation stability rankings are not fixed model traits but depend entirely on the attribution operator used.
True-Positive Filtering: By filtering for samples where predictions remained correct, the authors isolated explanation evolution from error correction, proving that instability arises from internal representation reorganization even when accuracy is maintained.
Architectural Mechanism Analysis: The study links specific architectural features (dense connectivity, multi-scale pathways, residual shortcuts) to their susceptibility to method-dependent stability variations.

Significance and Claims
The paper claims that the field cannot justify general explanation stability claims from single-method evaluations. The authors argue that:

Stability is a Pair Property: Explanation stability is an emergent property of the model–method pair, not an intrinsic model trait.
Regulatory Necessity: Regulatory submissions for medical AI must explicitly specify the attribution operator. Claims of safety or reliability based on a single method are insufficient and potentially misleading.
Evaluation Reform: XAI benchmarks and research must adopt cross-method validation. Reporting results under only one paradigm (e.g., LayerCAM) creates a false sense of robustness, as demonstrated by InceptionV3's high LayerCAM stability masking its fragility under Grad-CAM++.
Architectural Selection: For explanation-critical deployments, architectures should be selected based on cross-method robustness (e.g., DenseNet201) rather than predictive accuracy alone, as equivalent accuracy can mask fundamental differences in explanation reliability.

The authors conclude that explanation-based claims require the same methodological rigor as predictive evaluation, necessitating explicit conditioning on the attribution paradigm used.

The Great Highlighter Swap

The Plot Twist: It Depends on the Tool

Why This Matters: The "Illusory Safety" Trap

The Takeaway: Check Your Tools

Technical Summary: Explanation Stability Is a Property of the Model–Method Pair, Not the Model

More like this