🤖 machine learning

Addressing divergent representations from causal interventions on neural networks

This paper demonstrates that common causal interventions in neural networks often create out-of-distribution representations that can lead to misleading explanations, and proposes a modified Counterfactual Latent loss to mitigate these "pernicious" divergences while preserving interpretive power.

Original authors: Satchel Grant, Simon Jerome Han, Alexa R. Tartaglini, Christopher Potts

Published 2026-04-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Satchel Grant, Simon Jerome Han, Alexa R. Tartaglini, Christopher Potts

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand how a complex machine, like a giant, self-driving robot, makes decisions. You can't just look at the outside; you have to peek inside its "brain" (the neural network) to see what's happening.

A popular way to do this is called Causal Intervention. Think of it like a mechanic swapping out a specific part of the robot's brain with a part from a different robot to see if the behavior changes. If the robot suddenly starts driving to the beach instead of the grocery store, you know that specific brain part was responsible for "going to the store."

This paper, titled "Addressing Divergent Representations from Causal Interventions," argues that while this mechanic's method is powerful, it has a hidden flaw: The parts you swap in often don't fit the new machine properly.

Here is a breakdown of the paper's main ideas using simple analogies:

1. The Problem: The "Frankenstein" Brain

When researchers swap a piece of the robot's brain (an internal representation) with a piece from a different context, they often create a "divergent representation."

The Analogy: Imagine you are baking a cake. You take a perfectly baked sponge cake (the natural brain state) and try to replace a slice with a slice of raw dough from a different recipe (the intervention).
The Result: The cake still looks like a cake, but that one slice is raw, weird, and doesn't belong there. It's "out of distribution."
The Risk: The paper shows that when we do these swaps in AI, the new "slice" often doesn't fit the natural flow of the machine. It's like forcing a square peg into a round hole. The machine might still work, but it's working in a way it was never designed to do.

2. Two Types of "Bad Fits"

The authors explain that these "bad fits" (divergences) can be either harmless or dangerous.

A. The Harmless Glitch (The "Null-Space" Divergence)

Sometimes, the weird slice of raw dough is in a part of the cake that no one eats.

The Analogy: Imagine the robot has a hidden compartment in its brain that controls the color of its blinking lights, but the lights are currently off. If you swap in a weird signal there, the lights stay off. The robot's behavior (driving) doesn't change.
The Takeaway: If the weird signal is in a part of the brain that doesn't affect the final decision, it's harmless. It's just background noise.

B. The Pernicious Glitch (The "Hidden Pathway" Divergence)

Sometimes, the weird slice of raw dough triggers a secret mechanism that was supposed to stay asleep.

The Analogy: Imagine you swap a part of the robot's brain, and suddenly, a hidden lever gets pulled. The robot still drives to the store (so you think the experiment worked), but now it's also secretly activating a "self-destruct" mode or a "dance mode" that only triggers when the brain is in this weird, unnatural state.
The Danger: You might think, "Great! We found the part that controls driving to the store!" But actually, you accidentally woke up a dormant, dangerous pathway. You are misinterpreting how the robot naturally works because you are testing it in an unnatural, broken state.

3. The Solution: The "Fit-Check" Loss

The authors propose a fix to stop these "Frankenstein" brains from forming. They introduce a new rule for their experiments called the Counterfactual Latent (CL) Loss.

The Analogy: Before you swap that raw dough slice into the cake, you run it through a "Fit-Check" machine. This machine asks: "Does this slice look like it belongs in a normal cake?"
How it works: If the slice looks too weird (too far from the natural distribution), the machine forces it to morph slightly so it fits the cake's texture better, without changing the flavor (the causal meaning).
The Result: The researchers tested this on a large language model (like the AI you are talking to now). By using this "Fit-Check," they were able to swap brain parts without creating those dangerous, weird "raw dough" slices. The robot still learned the right lessons, but it didn't accidentally wake up hidden, dangerous pathways.

Why Does This Matter?

For a long time, scientists thought, "If the robot behaves correctly after the swap, the experiment is valid."

This paper says: "Not so fast."
If the robot behaves correctly only because you accidentally triggered a secret, weird pathway, your explanation is wrong. You might think you understand the robot's natural brain, but you've actually just built a weird, broken version of it.

In short: To truly understand AI, we need to make sure our experiments don't break the machine in the process. We need to swap parts in a way that keeps the machine feeling "natural," ensuring our discoveries are real and not just illusions caused by broken parts.

1. Problem Statement

Mechanistic interpretability relies heavily on causal interventions (e.g., activation patching, Distributed Alignment Search) to determine what internal neural representations encode. These methods manipulate internal states to observe changes in model behavior.

The core problem identified by the authors is representational divergence:

Intervened representations often lie out-of-distribution (OOD) relative to the model's natural latent space.
Standard assumptions in interpretability presume that counterfactual states created by interventions are "realistic" or faithful to the model's natural operation.
The paper argues that this assumption is frequently untested and often false. When interventions push representations into regions the model has never seen naturally, they may activate hidden pathways or dormant behaviors, leading to misleading conclusions about the model's true mechanisms.

2. Methodology

The authors employ a combination of theoretical proofs, empirical analysis on Large Language Models (LLMs), and synthetic experiments to characterize and mitigate divergence.

A. Theoretical Analysis of Divergence

Coordinate Patching Proof: The authors prove that for most manifold geometries (excluding axis-aligned hyperrectangles), coordinate-based patching (swapping specific vector dimensions between source and target) mathematically guarantees that the resulting vector will fall off the natural manifold.
Classification of Divergence: They categorize divergences into two types:
1. Harmless Divergence: Occurs within the behavioral null-space of the network layers. These changes do not affect the final output or downstream computations (e.g., $Wv = 0$).
2. Pernicious Divergence: Occurs when interventions activate hidden pathways (units/circuits silent in natural data) or trigger dormant behavioral changes (behavioral flips that only occur under specific, unseen contexts).

B. Empirical Validation

Datasets & Models: Experiments were conducted on Meta-Llama-3-8B-Instruct and synthetic Multi-Layer Perceptrons (MLPs).
Intervention Methods Tested:
- Mean Difference Vector Patching.
- Sparse Autoencoders (SAE) reconstructions.
- Distributed Alignment Search (DAS).
Metrics: The authors used Earth Mover's Distance (EMD) and Local PCA Distance to quantify the divergence between natural and intervened distributions.

C. Mitigation Strategy: Counterfactual Latent (CL) Loss

To address pernicious divergence, the authors propose minimizing the distance between intervened representations and the natural manifold.

CL Vectors: Defined as natural vectors that possess the same causal variable values as the intended intervened state.
CL Loss: An auxiliary loss function combining L2 distance and cosine similarity between the intervened vector ( $\hat{h}$ ) and the CL vector ( $h_{CL}$ ).
$L_{CL} = \frac{1}{2}\|\hat{h} - h_{CL}\|_2^2 - \frac{1}{2} \frac{\hat{h} \cdot h_{CL}}{\|\hat{h}\|_2 \|h_{CL}\|_2}$
Modified CL Loss: A variant that applies the loss specifically to causal subspaces (identified via DAS) rather than the entire vector, allowing for targeted constraint of the intervention dimensions.

3. Key Contributions

Demonstration of Ubiquity: Proved theoretically and empirically that divergent representations are a common, almost guaranteed outcome of standard causal intervention techniques across various methods (patching, SAEs, DAS).
Taxonomy of Harm: Provided a theoretical framework distinguishing between harmless (null-space) and pernicious (hidden pathway activation) divergences. They show that "safe" behavior in one context does not guarantee safety in others due to dormant behavioral changes.
Mitigation via CL Loss:
- Applied the Counterfactual Latent loss to Boundless DAS on a 7B LLM, demonstrating that divergence (EMD) can be significantly reduced while maintaining or improving Interchange Intervention Accuracy (IIA).
- Introduced a Modified CL Loss targeting causal subspaces, which improved Out-of-Distribution (OOD) generalization in synthetic tasks.
Correlation with Performance: Established a strong negative correlation between representational divergence (EMD) and OOD intervention performance, suggesting that minimizing divergence is a proxy for improving the reliability of interpretability claims.

4. Key Results

Empirical Divergence: In experiments with Llama-3-8B, intervened representations consistently showed higher EMD and Local PCA distances compared to natural representations, confirming that interventions often push the model off-manifold.
Hidden Pathways: Synthetic examples demonstrated that mean-difference patching could flip a decision by activating a ReLU unit that is never active in natural data, leading to a "misleadingly confirmatory" result.
Dormant Changes: Interventions that appeared benign in one context caused catastrophic behavioral flips in another context (e.g., switching from Class A to Class C) purely because the intervention primed a dormant pathway.
Effectiveness of CL Loss:
- In the Boundless DAS setting, increasing the weight of the CL loss ( $\epsilon$ ) reduced EMD (divergence) without sacrificing IIA.
- In synthetic OOD tasks, models trained with the Modified CL Loss achieved significantly higher accuracy on held-out partitions compared to those trained with behavioral loss alone.
- Regression analysis showed a statistically significant negative correlation ( $R^2 = 0.73$ ) between training EMD and OOD IIA.

5. Significance and Implications

Reliability of Interpretability: The paper challenges the validity of many current mechanistic interpretability claims. If an intervention relies on a divergent representation, the observed behavior may be an artifact of the intervention method rather than a property of the model's natural circuitry.
Methodological Shift: The work advocates for manifold-preserving interventions. It suggests that future interpretability methods must explicitly constrain interventions to remain within the natural distribution of the model.
Practical Solution: The proposed CL loss offers a practical, trainable mechanism to align interventions with natural representations, serving as a step toward more robust and faithful mechanistic explanations.
Future Directions: The authors note that while minimizing magnitude helps, it does not fully eliminate the risk of hidden pathways. Future work should focus on principled methods to classify and detect pernicious divergence specifically, rather than just minimizing distance.

In summary, this paper identifies a critical flaw in current causal intervention practices (divergence from natural distributions) and provides both a theoretical framework to understand the risks and a practical loss function (CL Loss) to mitigate them, thereby advancing the reliability of mechanistic interpretability.