🤖 machine learning

The Curse of Multiple Mediators: Hidden Interaction Effects in Activation Patching

This paper reveals that activation patching estimates in mechanistic interpretability inherently conflate natural indirect effects with interaction effects dependent on other components' states, arguing that rather than eliminating this interaction, researchers should leverage it as a diagnostic tool to identify prompt-dependent causal mechanisms and the limitations of greedy component ranking.

Original authors: Sankaran Vaidyanathan, David Arbour, Aaron Mueller, Scott Niekum, David Jensen

Published 2026-06-29

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Sankaran Vaidyanathan, David Arbour, Aaron Mueller, Scott Niekum, David Jensen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Solo Act" Illusion

Imagine you are trying to figure out which musician in a band is responsible for a specific part of a song. You have a magical tool called Activation Patching. This tool works like a "time-travel mute button."

To test a specific musician (let's call him the Guitarist), you:

Record the band playing the song perfectly (Clean Input).
Record the band playing a slightly messed-up version of the song (Corrupted Input).
The Patch: You take the Guitarist's performance from the perfect recording and paste it into the messed-up recording.
The Result: If the song sounds better, you assume the Guitarist is the hero. If it sounds the same, you assume the Guitarist doesn't matter.

For years, researchers have used this method to rank which parts of AI models (neural networks) are important. They call this measurement the NIE (Natural Indirect Effect).

The Problem: The "Hidden Handshake"

The paper argues that this "Solo Act" test is flawed because it ignores Interaction Effects (INT).

In a real band, musicians don't just play alone; they listen to each other. The Guitarist might play a solo that only sounds good if the Drummer is keeping a steady beat. If the Drummer is messing up, the Guitarist's solo might sound terrible, even if the Guitarist is talented.

In the AI model, there is a "backdoor" called the Residual Stream. It's like a side channel that lets information skip over the component you are testing.

When you test the Guitarist, you fix his notes (Clean) but leave the rest of the band playing the messed-up version (Corrupted).
The Guitarist's "clean" notes try to mix with the Drummer's "messed-up" rhythm.
Because the model is complex and non-linear (like a chaotic jam session), the result isn't just "Guitarist + Drummer." It's a weird, unpredictable collision of the two.

The paper proves that the score you get (NIE) is actually a mix of two things:

PIE (Pure Indirect Effect): How good the Guitarist is on his own.
INT (Interaction Effect): How much the Guitarist's performance depends on the rest of the band.

The Catch: The standard test (NIE) gives you the sum of both, but it doesn't tell you which is which.

The Three Scenarios (What the Paper Found)

The authors tested this on a famous AI task called IOI (Indirect Object Identification), where the AI has to guess who did what to whom in a sentence like "John gave the book to Mary." They found three distinct ways this "interaction" messes up the rankings:

1. The "Backup Plan" (Hidden Heroes)

The Situation: Imagine the Guitarist is the main soloist, but there is a Backup Guitarist who plays the exact same notes.
The Flaw: When you test the Backup Guitarist alone, the main Guitarist is still playing the messed-up version. The Backup Guitarist tries to fix it, but the main Guitarist's bad rhythm ruins the Backup's attempt.
The Result: The Backup Guitarist gets a terrible score (NIE) because the interaction is negative. The test says, "This musician is useless!"
Reality: The Backup Guitarist is actually vital. If you remove the Main Guitarist, the Backup saves the day. But the standard test hides this because it doesn't account for the "teamwork" required to see their value.

2. The "Context Specialist" (The Silent Partner)

The Situation: Imagine a Musician who only plays a specific note if the Drummer is playing a specific beat.
The Flaw: When you test this Musician alone, the Drummer is playing the wrong beat. The Musician's special note doesn't trigger because the condition isn't met.
The Result: The Musician gets a score of zero. The test says, "This musician does nothing."
Reality: This Musician is essential, but only when the rest of the band is working correctly. The standard test misses them entirely because it isolates them from the context they need to shine.

3. The "Saboteur" (The Harmful Player)

The Situation: Imagine a Musician who plays a terrible note that ruins the song, but the rest of the band has a special "noise-canceling" trick that fixes it.
The Flaw: When you test this Musician alone, the rest of the band is messed up, so they can't use their noise-canceling trick. The Musician's bad note destroys the song.
The Result: The Musician gets a huge negative score.
Reality: This Musician is actually part of a complex system where their "bad" note is intentionally countered by others. The test sees only the destruction, not the balance.

Why This Matters (The "So What?")

The paper makes three main points:

Rankings are Wrong: If you rank AI components by the standard score (NIE), you will miss the "Backup" heroes and the "Context Specialists." You might think the AI works a certain way, but you're actually looking at a distorted picture.
It's Not a Bug, It's a Feature: The authors argue we shouldn't try to "fix" the math to remove these interactions. Instead, we should measure them.
- If the interaction score is negative, it means the component is a "Backup" (it helps when others fail).
- If the interaction score is positive, it means the component is a "Context Specialist" (it needs others to work).
The "Prompt" Problem: The paper shows that these interaction scores change wildly depending on the specific sentence (prompt) you use. If you change the sentence slightly, the "Backup" might stop being a backup, and the "Specialist" might stop working. This explains why AI circuit studies often give inconsistent results.

The Conclusion

The paper concludes that Activation Patching (the standard tool) is like looking at a puzzle piece in isolation. You can see the shape, but you can't see how it fits with the neighbors.

The "Interaction Effect" (INT) is the missing piece of information. It tells us that in complex AI models, no component works alone. To truly understand how an AI thinks, we have to stop asking "What does this part do?" and start asking "What does this part do when the rest of the model is doing X?"

The authors provide a new way to calculate these interactions, allowing researchers to finally spot the "Backup" mechanisms and "Context Specialists" that were previously invisible, turning a confusing mess of data into a clearer map of how the AI actually works.

Technical Summary: The Curse of Multiple Mediators: Hidden Interaction Effects in Activation Patching

1. Problem Statement

Activation patching is the de facto standard in mechanistic interpretability for attributing causal responsibility to individual neural network components. It operationalizes the Natural Indirect Effect (NIE) from causal mediation analysis by substituting a component's activations with counterfactual values (typically from a "clean" input) and measuring the resulting change in output.

However, the paper identifies a fundamental flaw in this approach when applied to transformers: the NIE does not solely capture the causal effect flowing through a specific component. Due to the transformer's residual stream architecture, which provides bypass paths (skip connections) around every component, the NIE conflates the component's direct contribution with interaction effects (INT). These interactions measure how a component's causal effect depends on the state of other components (the bypass). Consequently, standard activation patching rankings can be misleading, causing functionally important "backup" mechanisms to appear invisible (false negatives) or artificially inflating the importance of others.

2. Methodology

The authors re-derive the activation patching estimand from the perspective of causal mediation analysis, specifically addressing the structural condition of "recanting witnesses" inherent in transformer skip connections.

Theoretical Decomposition: The paper mathematically decomposes the standard "noising" estimator (the default activation patching method) into two distinct causal quantities:
1. Pure Indirect Effect (PIE): The path-specific effect flowing exclusively through the target component, effectively blocking all bypass paths.
2. Interaction Effect (INT): The mediated interaction between the component and its bypass.
  The relationship is defined as: $NIE = PIE + INT$.
  The paper also defines a "denoising" estimator, which the authors show computes the PIE directly, contrasting it with the NIE computed by noising.
Theoretical Analysis:
- Second-Order Approximation: The authors prove that INT scales with the distance between clean and patched activations. Specifically, INT is approximated by the mixed second directional derivative (Hessian bilinear form) of the downstream function with respect to perturbations in the component and its bypass. INT vanishes if the downstream map is locally affine or if the perturbations are zero.
- Multi-Component Decomposition: Using inclusion-exclusion principles, the authors extend the decomposition to sets of components. They show that the total effect of patching a set of nodes decomposes into the sum of individual PIEs, individual INTs, and cross-component interactions ($xINT$) between subsets of nodes.
Empirical Validation:
- Datasets: Experiments utilize the GPT-2 IOI (Indirect Object Identification) circuit, along with Gender Bias, Greater Than, and SVA tasks.
- Models: GPT-2 Small, Pythia-70M, and Qwen2.5-0.5B.
- Metrics: The study compares rankings based on NIE versus PIE, analyzes the variance of faithfulness scores, and examines the magnitude of INT across different functional head groups (e.g., Name Movers, Induction heads).

3. Key Contributions and Results

A. Divergence of NIE and PIE Rankings

The paper demonstrates that NIE and PIE induce substantially different rankings of component importance.

Rank Correlation: On the GPT-2 IOI task, the Spearman rank correlation between NIE and PIE is as low as $\rho = 0.51$ .
Prompt Dependence: The divergence is driven by the semantic distance between clean and counterfactual prompts. Tasks with semantically close prompts (e.g., Gender Bias) show high correlation ( $\rho \approx 0.99$ ), while tasks with larger perturbations (e.g., IOI with pABC corruption) show low correlation.

B. The Role of Interaction Effects (INT)

Backup Mechanisms: In the IOI circuit, "Backup Name Mover Heads" (BNMH) have high PIE scores but are suppressed in NIE rankings due to negative INT. Their contribution is conditional on the primary Name Mover Heads being ablated; when the primary heads are active (clean input), the backup heads' marginal contribution shrinks, resulting in a negative interaction term that lowers their NIE score.
Context-Specific Mechanisms: Heads like Duplicate Token (DTH), Induction, and Previous Token (PTH) have near-zero PIE scores because their contribution relies entirely on the presence of other active components. Consequently, they are invisible to PIE-based ranking but are captured by NIE (which includes the positive INT).
Harmful Mechanisms: Negative Name Mover Heads (NNMH) exhibit negative PIE (harming performance) but positive INT (partially opposed by other heads), illustrating how INT can mask harmful components.

C. Explaining Faithfulness Instability

The paper provides a mechanistic explanation for the high variability of "faithfulness scores" (metrics evaluating how well a circuit recovers model performance).

Cross-Interaction Variance: Faithfulness scores decompose into PIE, individual INT, and cross-component INT ($xINT$). The authors find that $xINT$ terms exhibit significantly higher variance across different prompt templates (e.g., ABBA vs. BABA) than PIE terms.
Magnitude: In the IOI task, the sum of individual PIEs predicts a much larger effect than the actual circuit performance. The discrepancy is explained by large negative cross-interactions ($xINT$), indicating that the heads collectively produce less than the sum of their individual parts due to nonlinear coupling.

D. Limitations of Remedies

The authors argue that simply removing INT to rank by PIE introduces new failure modes. While PIE correctly identifies backup mechanisms, it fails to identify context-specific mechanisms (like Induction heads) that are essential for the circuit to function but have zero path-specific effect. Thus, neither NIE nor PIE alone provides a complete picture.

4. Significance and Claims

The paper claims that interaction effects are not a nuisance to be eliminated but a diagnostic feature of causal mediation in transformers.

Unifying Theory: The presence of INT offers a unifying explanation for several known pathologies in mechanistic interpretability, including the failure of greedy component ranking, the instability of faithfulness scores across prompts, and the difficulty of discovering backup mechanisms without combinatorial search.
Reframing Interpretability: The authors argue that INT is a verifiable statement of the assumptions of activation patching. It signals when causal conclusions are prompt-dependent and when greedy ranking will miss mechanisms discoverable only through combinatorial search.
Practical Utility: While INT does not eliminate the need for combinatorial search to resolve all pathologies, it is measurable at various granularities. Practitioners can use the magnitude and sign of INT to identify when their conclusions are trustworthy without exhaustive search, or when they are likely missing critical context-dependent or backup mechanisms.

The paper concludes that the "Curse of Multiple Mediators" is an inherent property of transformer architecture and causal mediation, requiring a shift from viewing components in isolation to understanding their interactive, state-dependent roles.