🤖 AI

Activation-Deactivation: A General Framework for Robust Post-hoc Explainable AI

This paper introduces Activation-Deactivation (AD), a novel post-hoc explainability framework that replaces input perturbations with internal model deactivation to generate more robust and transferable explanations without requiring additional training.

Original authors: Akchunya Chanchal, David A. Kelly, Hana Chockler

Published 2026-07-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Akchunya Chanchal, David A. Kelly, Hana Chockler

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, but mysterious, robot that looks at pictures and tells you what it sees (like "That's an ibex!" or "That's a platypus!"). The problem is, the robot is a "black box"—you can't see how it thinks. You want to know: Which parts of the picture made the robot say "ibex"?

This is the problem of Explainable AI (XAI). The paper introduces a new way to answer this question called Activation-Deactivation (AD), and a specific tool for it called convad.

Here is how it works, using simple analogies:

The Old Way: The "Bad Neighbor" Problem

Currently, most methods try to explain the robot's decision by mutating the picture.

The Analogy: Imagine you want to know if a chef's famous soup tastes good because of the carrots. To test this, you take a spoonful of the soup and replace the carrots with... toilet paper, sand, or blue paint.
The Problem: If you put toilet paper in the soup, the soup is no longer "soup." It's a weird, out-of-distribution mess. If the chef says, "I don't like this," you don't know if it's because the carrots were missing, or just because you ruined the soup with toilet paper.
In the Paper: This is called using "out-of-distribution mutants." The old methods mask parts of the image with random values (like zero, gray, or noise). Depending on what you mask with, you get different, often confusing answers. Sometimes the robot thinks a picture of a snowman is a sheep just because you masked the snow with the wrong color.

The New Way: The "Silent Switch" (Activation-Deactivation)

The authors say: "Why change the ingredients? Just tell the robot to ignore them."

The Analogy: Instead of replacing the carrots with toilet paper, imagine you simply turn off the lights in the kitchen section where the carrots are. The carrots are still there, but the chef's eyes (the robot's sensors) can't see them. The rest of the soup remains perfectly normal.
How it works: The convad tool doesn't change the picture at all. Instead, it goes inside the robot's brain (the neural network) and flips switches. If a part of the picture is supposed to be "masked," the tool stops the information from that part of the picture from traveling through the robot's brain. It effectively says, "Pretend this part of the image doesn't exist," without actually changing the image data.

Why is this better?

No Guessing Game: Old methods require you to guess: "Should I mask with black? Gray? White?" The paper shows that the answer changes completely depending on your guess. convad needs no guess. It just turns the signal off.
Stability: Because it doesn't introduce weird "toilet paper" pixels, the robot's reaction is much more reliable. The paper tested this on many different types of robots (models) and pictures (datasets).
The "Goldilocks" Zone: The researchers found that explanations from convad are just right. They aren't too big (wasting space on irrelevant pixels), they aren't too noisy (confusing), and they are very robust (they still work even if you put the picture against a plain background).

The Results

The paper tested convad against the current best methods (like LayerMask, LIME, and Grad-CAM).

Robustness: convad was 30-40% better at giving explanations that the robot actually trusted, even when the background changed.
Transferability: If you find the "important parts" of a picture using convad on one type of robot, those same parts usually work for a different type of robot. Other methods often fail at this.
No Training Needed: You can take any robot that is already trained, plug convad in, and it works immediately. You don't have to re-teach the robot anything.

The Catch (Limitations)

Inside Access: To use convad, you need to be able to open the robot's brain and touch the wires (access the model's internal layers). You can't use it on a robot you can't see inside (a true "black box").
Leakage: Sometimes, if the "off" switch isn't perfect, a tiny bit of information might "leak" through, like a light shining under a door. The authors acknowledge this but say it's a rare edge case.

Summary

The paper proposes a new way to understand AI decisions. Instead of messing up the input picture with strange values (like the "toilet paper" soup), convad simply tells the AI to ignore specific parts of the picture by cutting off the signal inside the model. This leads to clearer, more reliable, and more trustworthy explanations of why an AI made a decision.

Technical Summary: Activation-Deactivation (AD) for Robust Post-hoc Explainable AI

1. Problem Statement

Current perturbation-based Explainable AI (XAI) methods, such as LIME, SHAP, and REX, rely on creating "mutants" of the input data by masking or occluding specific features. These methods typically replace masked pixels with predefined values (e.g., zero, mean, Gaussian blur, or random noise). The authors argue that this approach generates out-of-distribution (o.o.d.) inputs, which cast doubt on the faithfulness of the resulting explanations.

The choice of masking value significantly impacts the explanation quality. As demonstrated in the paper, different masking values (e.g., zero vs. max) can lead to vastly different minimal sufficient pixel sets (MSPS), some of which may not align with human intuition or the model's actual decision boundaries. Furthermore, existing methods often require domain-specific adaptations or expert knowledge to select the "correct" masking value, introducing subjectivity and inconsistency.

2. Methodology: Activation-Deactivation (AD)

The paper introduces Activation-Deactivation (AD), a novel forward-pass paradigm that eliminates the need for input perturbation entirely. Instead of modifying the input image, AD modifies the model's internal state during inference.

Core Mechanism:

Concept: AD "switches off" parts of the model corresponding to intended perturbations. It restricts the propagation of activations associated with occluded input features, effectively forcing the model to disregard those regions without altering the input distribution.
Implementation (convad): The authors implement AD for Convolutional Neural Networks (CNNs) via an algorithm named convad.
- Checkpoints: The algorithm inserts checkpoints at specific layers (e.g., prior to dimensionality-altering operations or between convolutional layers).
- Position-Attribution: It utilizes a position-attribution function, $\Phi_i$ , to calculate the ratio of masked to unmasked features contributing to a specific activation.
- Thresholding: A hyperparameter $\tau$ determines the threshold for deactivation. If the ratio of masked features exceeds $\tau$ , the corresponding activation is set to zero.
- Mask Propagation: The mask is updated and propagated through the network. For shape-preserving operations, a mean-value kernel convolution is used to update the mask. For dimensionality-altering operations, the mask is adjusted based on the specific operation (e.g., padding with 1s or 0s).

Theoretical Guarantees:
The paper proves that if the input mask is "empty" (all ones), the convad-instrumented model $N'$ behaves identically to the original model $N$ . This ensures that the framework does not alter the model's baseline behavior when no features are occluded, a guarantee the authors note is lacking in similar tools like LayerMask.

3. Key Contributions

Novel Paradigm: Introduction of AD, which replaces input perturbation with internal activation deactivation, thereby keeping the model's input features strictly in-distribution.
Algorithm Implementation: Development of convad, a "drop-in" mechanism for trained CNNs that requires no additional training, fine-tuning, or domain-specific knowledge of masking values.
Theoretical Framework: Formal definitions for position functions and position-attribution functions within the AD framework, along with proofs of behavioral equivalence to the original model under unmasked conditions.
Comprehensive Evaluation: A rigorous comparison against state-of-the-art (SOTA) black-box and white-box methods (LIME, Grad-CAM, REX) and the LayerMask tool across multiple architectures (ResNet-50, RegNetY-12GF, EfficientNet-V2) and datasets (ImageNet-1k, ImageNet-v2, PASCAL-VOC).

4. Experimental Results

The authors evaluated explanation quality based on robustness, transferability, size, redundancy, and overlap.

Robustness: convad consistently outperformed all masking-based strategies. On solid-color backgrounds, convad explanations retained the correct classification significantly more often. On in-distribution (i.i.d.) backgrounds, convad showed approximately 20% robustness at high confidence thresholds, outperforming other methods by 30–40% on average.
Transferability: MSPSs generated by convad demonstrated up to 22% better transferability across different model architectures compared to other methods, suggesting they rely on truly discriminative features rather than shortcuts.
Goldilocks Zone: Analysis of the relationship between robustness, explanation size, and noise revealed that convad explanations occupy a "Goldilocks" zone: they are highly robust, have low noise (redundancy), and are compact in size. Other methods tended to fail in at least one of these dimensions.
Independence from Masking Values: Unlike perturbation methods where results vary drastically based on the chosen masking value (e.g., Zero vs. Max), convad produced consistent results without requiring a choice of occlusion value.

5. Significance and Claims

The paper claims that AD addresses the fundamental flaw of perturbation-based XAI: the generation of out-of-distribution inputs. By deactivating activations rather than mutating inputs, convad provides explanations that are:

More Robust: Less sensitive to the specific choice of perturbation strategy.
More Transferable: Features identified as important are consistent across different model architectures.
Model-Agnostic (within architecture): The algorithm is the same for all models of the same architecture (CNNs), requiring no retraining.
Faithful: The explanations adhere strictly to the model's confidence thresholds and decision logic without introducing artificial artifacts via masking values.

The authors conclude that while the current implementation is limited to CNNs and requires white-box access to modify internal layers, the AD framework offers a principled alternative to masking that significantly improves the reliability of post-hoc explanations. Future work is planned to extend this framework to transformers and attention-based models.