What is Missing? Explaining Neurons Activated by Absent Concepts

This paper identifies that deep neural networks frequently encode the absence of concepts to drive neuron activation—a phenomenon largely overlooked by standard explainable AI methods—and proposes simple extensions to attribution and feature visualization techniques to effectively reveal and leverage these "missing" concepts for better model interpretation and debiasing.

The Gist

Here is an explanation of the paper "Explaining Neurons Activated by Absent Concepts," broken down into simple language with creative analogies.

The Big Idea: It's Not Just What You See, It's What You Don't See

Imagine you are a detective trying to solve a mystery. Usually, when we ask an AI (a computer brain) "How did you solve this?", the AI points to the clues it found.

• The AI says: "I saw a red hat, so I know it's a clown."
• The Reality: The AI might actually be thinking, "I saw a red hat, AND I didn't see a police badge, so it's definitely a clown."

This paper argues that current tools for explaining AI are like a detective who only looks at the clues that are present. They completely ignore the clues that are missing. The authors call these missing clues "Encoded Absences."

---

1. The Problem: The "Missing Clue" Blind Spot

In the world of Artificial Intelligence (specifically Deep Neural Networks), we use "Explainable AI" (XAI) tools to understand how the computer makes decisions.

• Standard Tools: These tools highlight the pixels in an image that made the AI say "Yes." If you show a picture of a dog, the tool highlights the ears and the tail.
• The Flaw: These tools assume that if a feature isn't highlighted, it doesn't matter. But sometimes, the absence of a feature is the most important part of the decision.

The Analogy: The "No Smoking" Sign
Imagine a bouncer at a club.

• Standard Explanation: The bouncer says, "I let you in because you have a VIP pass." (This is the presence of a concept).
• The Hidden Logic: The bouncer actually let you in because you didn't have a "No Entry" sticker on your forehead. If you did have that sticker, you would have been kicked out.
• The AI's Mistake: Current AI tools only show the VIP pass. They don't show that the lack of the "No Entry" sticker was the real reason you got in.

2. How the AI "Thinks" About Missing Things

The authors show that AI models are smart enough to learn this "negative logic." They don't just learn "Dog = Ears + Tail." They learn "Dog = Ears + Tail + NO Cat Ears."

The Biological Example: The Fly's Brain
The paper mentions a fly's eye. A fly has a neuron that fires when it sees something moving to the right. But it only fires if there is no movement to the left. If something moves left, the neuron shuts down. The fly's brain encodes the absence of leftward motion to know it's safe to fly right.

The AI Example: Irish Setters vs. Spaniels
If an AI is trying to tell the difference between an Irish Setter dog and a Sussex Spaniel, it might look for the Setter's long ears. But to be sure, it also checks: "Is there a Spaniel's short snout?" If the snout is missing, the AI gets even more confident it's a Setter.

3. Why Current Tools Fail

The paper explains that standard AI explanation tools are like a flashlight that only shines on things that are there.

• Feature Visualization: This tool tries to create an image that makes a specific neuron fire as hard as possible. If a neuron fires when a "Cat" is absent, the tool tries to make an image with no cat. But it ends up just showing a blank wall or a generic background. It fails to tell you what is missing.
• Attribution Maps: These highlight the pixels that contributed to a decision. If a decision was made because a "Cat" was missing, the tool can't highlight a missing cat. It just highlights the dog that is there, missing the whole point.

4. The Solution: The "Reverse Flashlight"

The authors propose two simple tricks to fix this:

Trick A: The "Non-Target" Attribution
Instead of asking, "What made the AI say 'Dog'?", we ask, "What would make the AI say 'Cat'?"

• We show the AI a picture of a Cat.
• We ask the AI to explain why it didn't say "Dog."
• The AI will point to the Cat features and say, "These features are bad for the 'Dog' prediction."
• Result: We now see the "negative clues" (the red highlights) that tell us the AI is looking for the absence of a cat to identify a dog.

Trick B: Feature Visualization via Minimization
Instead of asking the AI, "Show me what makes this neuron fire the most?", we ask, "Show me what makes this neuron fire the least?"

• If a neuron fires when a "Cat" is missing, the thing that makes it fire the least is an image full of Cats.
• Result: The tool generates an image of a Cat, revealing that the neuron is actually a "Cat Detector" that works by being silenced when a cat is present.

5. Why This Matters: Fixing Biased AI

The paper shows that this isn't just a theoretical curiosity; it's a real-world problem.

The Skin Cancer Example
Imagine an AI trained to spot skin cancer.

• The Bias: In the training data, "Benign" (safe) moles often had colorful patches on them (like a sticker). "Malignant" (cancerous) moles did not.
• The AI's Shortcut: The AI learned: "If I see a colorful patch, it's safe. If I don't see a colorful patch, it's cancer."
• The Danger: If you show the AI a cancerous mole with a colorful patch, it might get confused. If you show it a safe mole without a patch, it might think it's cancer.

The Fix:
The authors used their new "Reverse Flashlight" tools to see that the AI was relying on the absence of the patch to predict cancer. They then taught the AI to ignore both the presence and the absence of the patch. This made the AI much fairer and more accurate, because it stopped using the "sticker" as a shortcut.

Summary

• The Problem: AI tools only explain what is there, ignoring what is missing.
• The Discovery: AI models frequently make decisions based on what is not in the picture (e.g., "It's a dog because it's not a cat").
• The Fix: By flipping the questions (asking what makes the AI say "No" instead of "Yes"), we can reveal these hidden "missing" clues.
• The Benefit: This helps us understand AI better, spot hidden biases, and build smarter, fairer systems.

In short: To understand the AI, you have to listen to its silence as much as its noise.

Technical Summary

Here is a detailed technical summary of the paper "What is Missing? Explaining Neurons Activated by Absent Concepts" by Hesse et al.

1. Problem Statement

Current Explainable Artificial Intelligence (XAI) methods, particularly attribution (e.g., Integrated Gradients, Saliency Maps) and feature visualization, operate under a fundamental assumption: that a neuron's activation is primarily driven by the presence of specific concepts in the input.

• Attribution methods highlight input pixels that contribute positively to a prediction.
• Feature visualization generates inputs that maximize a neuron's activation.

The Gap: These methods largely ignore encoded absences—a causal relationship where the absence of a concept causes a neuron to activate (or conversely, the presence of a concept suppresses activation). In biological systems (e.g., the Hassenstein–Reichardt detector in flies) and fine-grained classification tasks, the absence of a competing feature is often a critical discriminative signal. Standard XAI fails to reveal these signals because:

1. Attribution maps only show features present in the input.
2. Feature visualization via maximization produces inputs that lack the inhibitory concept, failing to visualize what is missing.

2. Methodology

The authors propose a formal definition of encoded absence and introduce two simple modifications to standard XAI techniques to uncover them.

A. Formal Definition

Definition 2.1 (Encoded Absence): A neuron $z_j$ encodes the absence of a concept $\hat{x}$ if the presence of $\hat{x}$ in the input causes the activation of $z_j$ to decrease.
$$f^{(l)}_j([x, C_{\hat{x}}=1]) < f^{(l)}_j([x, C_{\hat{x}}=0])$$
This is modeled as an inhibitory relationship within a Structural Causal Model (SCM).

B. Proposed Extensions

To reveal these absences, the authors propose two modifications:

1. Non-Target Attribution:

   • Standard approach: Compute attribution for a target class $t$ using an input $x$ belonging to class $t$. This misses concepts that are absent in $t$ but present in other classes.
   • Proposed approach: Compute attribution for target class $t$ using inputs $x^{(c \neq t)}$ from other classes.
   • Mechanism: If a concept $\hat{x}$ is present in a non-target image but suppresses the activation for class $t$, the attribution for $\hat{x}$ will be negative. This negative attribution reveals the encoded absence.

2. Feature Visualization via Minimization:

   • Standard approach: Find input patterns that maximize neuron activation.
   • Proposed approach: Find input patterns that minimize neuron activation ($\hat{x} = \arg \min_x z_j(x)$).
   • Mechanism: Inputs that strongly suppress a neuron (leading to strong negative pre-activation values) highlight the specific concepts whose absence the neuron is designed to detect.

3. Key Contributions

1. Formalization: The paper provides the first systematic causal definition of "encoded absences" in Deep Neural Networks (DNNs), distinguishing them from simple non-overlapping features.
2. Mechanistic Proof: The authors provide a constructive proof showing how DNNs can implement encoded absences using negative weights (inhibition) combined with a positive potential (bias or another activating concept).
3. Methodological Innovation: They demonstrate that standard XAI fails to capture these relationships and propose the "Non-Target Attribution" and "Minimization-based Visualization" as necessary complements.
4. Empirical Validation: They validate these concepts across synthetic toy models, biological motion detectors, and large-scale ImageNet models.

4. Experimental Results

A. Synthetic & Toy Models

• Hassenstein-Reichardt Detector: A hand-crafted CNN designed to detect left-to-right motion by inhibiting right-to-left motion. Standard attribution only showed the presence of left-to-right motion. Non-target attribution correctly identified the negative attribution of right-to-left motion, proving the model encodes its absence.
• Toy Classification: A model trained to distinguish images with a green pixel vs. those without. Standard visualization for the "no green" class only highlighted non-green pixels. Minimization-based visualization revealed that the "green pixel" was the specific pattern that suppressed the neuron, confirming the encoding of absence.

B. ImageNet Models (ResNet-50, VGG19)

• Quantitative Analysis: The authors inserted "least activating" patches (found via minimization) into high-activation images. These patches caused a significant drop in channel activation compared to random patches, proving that ImageNet models systematically use encoded absences.
• Qualitative Analysis: In fine-grained classification (e.g., distinguishing a Border Collie from a Leonberger), channels important for the target class often encoded the absence of features specific to the competing class. This suggests models use "ruling out" logic for fine-grained distinctions.

C. Debiasing Application

• Scenario: Skin lesion classification (ISIC dataset) where benign samples spurious co-occur with colorful patches.
• Problem: Standard "presence debiasing" (suppressing attribution for the patch on benign samples) failed when the bias was inverted (malignant samples had patches). The model had learned to predict malignancy based on the absence of the patch.
• Solution: Presence + Absence Debiasing. By extending the attribution prior to include non-target attribution (suppressing the patch's influence on the malignant class prediction when the patch is present), the model was successfully debiased. This prevented the model from relying on either the presence or absence of the spurious feature.

5. Significance and Impact

• Completeness of Explanation: The paper argues that a complete explanation of a DNN must account for both what is present and what is missing. Ignoring absences leads to incomplete or misleading interpretations of model behavior.
• Robustness and Bias: Encoded absences are a common mechanism for fine-grained classification but also a source of subtle bias. Models may learn to rely on the absence of a feature (e.g., "no colorful patch = malignant") rather than the presence of a true pathological feature.
• Future Directions: The authors suggest that these techniques are applicable to Large Language Models (LLMs) to understand inhibitory relationships between tokens and to generative models to control what is not generated.

In conclusion, the paper establishes that absence is a causal factor in neural network decision-making. By adapting attribution and visualization to detect inhibitory signals, researchers can uncover a hidden layer of model logic that is critical for robustness, fairness, and accurate interpretation.