💬 NLP

Mechanistic Interpretability as Statistical Estimation: A Variance Analysis

This paper argues that mechanistic interpretability is fundamentally a statistical estimation problem plagued by high intrinsic variance in causal mediation scores, which causes circuit discovery pipelines to produce unstable and fragile results that necessitate a shift toward rigorous statistical robustness and stability reporting.

Original authors: Maxime Méloux, François Portet, Maxime Peyrard

Published 2026-06-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Maxime Méloux, François Portet, Maxime Peyrard

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Trying to Map a City with a Shaky Compass

Imagine you are trying to draw a map of a complex city (a neural network) to understand how it gets from Point A to Point B (how it solves a problem). Scientists in the field of Mechanistic Interpretability are like urban explorers. They want to find the specific "roads" and "intersections" (neurons and connections) that the city uses to make decisions. They call these special routes "circuits."

The paper argues that right now, these explorers are using a compass that spins wildly. They think they are finding the one true map, but in reality, they are drawing different maps every time they look, depending on tiny, random changes in the weather or how they hold the compass.

The Core Problem: The "Spinning Compass" (Variance)

The researchers discovered that the tools used to find these circuits are fundamentally unstable. They broke the problem down into three layers of instability:

1. The Raw Signal is Noisy (Intrinsic Variability)

The Analogy: Imagine trying to measure the strength of a specific wind current in a storm. If you measure it at one exact second, you get a number. If you measure it one second later, the wind might be slightly stronger or weaker because of random turbulence.
The Paper's Claim: The "importance" of a specific part of the AI isn't a fixed number like the weight of a rock. It's more like the wind speed. It changes wildly depending on the specific input (the "storm") you are looking at. The paper shows that even if you calculate this perfectly, the score for a single connection fluctuates so much that it's hard to tell if it's actually important or just a random gust.

2. The Tools Add More Noise (Approximation Errors)

The Analogy: Because measuring the wind perfectly takes too much time, the explorers use a cheap, fast sensor (approximation methods like EAP) to guess the wind speed. But this cheap sensor is jittery. It adds its own static and errors on top of the already changing wind.
The Paper's Claim: The fast methods scientists use to save time (like Edge Attribution Patching) introduce even more noise. They often make the scores look more unstable than they already were. The "signal-to-noise ratio" is so low that the tools are often just picking up static rather than the real signal.

3. The Map Changes with Every Step (Aggregation Instability)

The Analogy: To draw the final map, the explorers take 100 different measurements and average them out. But because the measurements are so shaky, if you change the list of 100 measurements just a little bit (like swapping one for another), the final map looks completely different. One day, the map shows a highway; the next day, it shows a dirt path.
The Paper's Claim: When scientists combine these shaky scores to build a "circuit," the result is incredibly fragile.

Data Changes: If they use a slightly different set of examples to train their map, they find a totally different circuit.
Method Changes: If they tweak a setting (like changing how they average the numbers), they find a different circuit.
Perturbation Changes: If they change how they "break" the AI to test it (the counterfactual), the circuit they find shifts again.

The "Dead Salmon" Effect

The paper mentions that current methods are prone to "dead salmon" artifacts.
The Analogy: In neuroscience, researchers once found "brain activity" in a dead salmon just because they didn't correct for statistical noise. The paper suggests AI researchers might be doing the same thing: finding "circuits" that look real but are actually just random noise that happened to line up by chance.

What the Researchers Found (The Evidence)

They ran experiments on different AI models (like GPT-2 and Llama) and tasks (like math or grammar).

The Result: When they ran the same experiment 100 times with slightly different data, the "circuits" they found barely overlapped. Sometimes, the overlap was less than 10%.
The Conclusion: There is no single "True Circuit" waiting to be discovered. Instead, there is a cloud of possible circuits, and current methods are just picking one random point from that cloud and calling it the truth.

The Proposed Solution: Stop Guessing, Start Measuring Uncertainty

The authors don't say "give up on AI interpretation." Instead, they say "stop pretending we know more than we do."

The New Rulebook:

Report the Variance: Don't just show one map. Show 100 maps and say, "Here is the average, and here is how much they differ."
Check Stability: Before claiming you found a circuit, check if it stays the same when you change the data slightly. If the circuit disappears with a tiny change, it's not reliable.
Embrace Uncertainty: Treat these findings like weather forecasts (e.g., "70% chance of rain") rather than absolute facts (e.g., "It will rain").

Summary

The paper argues that Mechanistic Interpretability is currently trying to do science without admitting it's dealing with statistics. It's like trying to measure the height of a mountain while standing on a trampoline. The authors want the field to stop looking for a single, perfect "ground truth" circuit and start reporting how shaky and variable their findings actually are. Only by acknowledging this instability can we trust the maps we are drawing of AI minds.

Technical Summary: Mechanistic Interpretability as Statistical Estimation: A Variance Analysis

Problem Statement

Mechanistic Interpretability (MI) aims to reverse-engineer deep neural networks by identifying "circuits"—functional sub-networks responsible for specific model behaviors. These circuits are typically discovered using Causal Mediation Analysis (CMA), which involves intervening on the computational graph to measure the causal effect of specific components (nodes or edges) on model outputs.

However, the scientific validity of these findings is increasingly questioned due to a lack of stability. Current MI practices often produce "point estimates" (single structural summaries) without quantifying uncertainty. The paper argues that circuit discovery is not a standalone task but a statistical estimation problem built upon CMA. The core problem identified is fundamental instability: the causal effect of a component is not a fixed property but a volatile random variable. Consequently, small perturbations in input data, hyperparameters, or intervention definitions yield vastly different circuits, undermining the reliability of MI findings and their generalizability.

Methodology

The authors frame circuit discovery as a downstream pipeline fueled by CMA and analyze its stability through a statistical lens, decomposing instability into two primary sources:

Variance (Sampling Sensitivity): Fluctuations arising from relying on finite datasets to approximate population expectations.
Robustness (Methodological Sensitivity): Sensitivity to the specification of counterfactuals (intervention strategies) and hyperparameters.

The study employs the following experimental setup:

Tasks: Three standard interpretability tasks: Indirect Object Identification (IOI), Subject-Verb Agreement (SVA), and Greater-Than.
Models: GPT-2-small, Llama-3.2-1B, and Llama-3.2-1B-Instruct.
Estimators: The study focuses on the Edge Attribution Patching (EAP) family, including exact CMA (where computationally feasible), EAP, and EAP-IG (with integrated gradients for inputs and activations).
Stability Metrics:
- Structural Stability: Measured via the pairwise Jaccard index of discovered edge sets.
- Functional Stability: Measured via Circuit Error (CE) and KL divergence between the full model and the isolated circuit.
Perturbation Regimes:
- Data Resampling: Bootstrap resampling to simulate limited sample sizes.
- Distribution Shifts: Using meta-datasets and prompt paraphrasing.
- Intervention Definition: Varying the noise amplitude in counterfactual generation (Gaussian noise on token embeddings).
- Methodological Perturbations: Varying aggregation functions (mean vs. median), patching types, and estimator choices.

Key Contributions and Results

1. Intrinsic Variability of CMA Scores

The authors demonstrate that even exact CMA scores exhibit high intrinsic variability. For a single edge, the causal effect is not constant across inputs drawn from the same distribution.

Finding: The coefficient of variation (CV) for exact edge scores is often close to 0.5.
Implication: The causal effect is a random variable dependent on the specific input-counterfactual pair, not a fixed property of the component.

2. Amplification by Approximation Methods

Approximation methods like EAP and EAP-IG introduce additional estimation noise on top of the intrinsic variance.

Finding: Approximate estimators often have a CV > 1, meaning the standard deviation of the score exceeds its mean. This results in a low signal-to-noise ratio, making the identification of stable circuits from finite samples statistically precarious.

3. Circuit Instability Across the Pipeline

The instability of edge scores propagates to the final circuit structure, leading to highly divergent results under minor changes:

Data Resampling: Bootstrap resampling yields the lowest structural consistency (mean Jaccard index $\approx$ 0.56 for GPT-2-small) and highest variability. Circuits discovered from resampled data are often structurally disjoint and less faithful to the original model.
Model Size: Stability degrades significantly in larger models (Llama-3.2-1B) compared to GPT-2-small, suggesting that identifying reliable circuits in more capable models is non-trivial.
Methodological Sensitivity: Changing hyperparameters (e.g., aggregation from sum to median, or patching type) can drop Jaccard similarity to near zero (e.g., 0.086), effectively returning a disjoint subgraph. Different EAP variants often isolate different artifacts of the high-variance distribution rather than converging on a single circuit.
Intervention Sensitivity: The identified circuit is not invariant to the magnitude of the perturbation. Varying the noise amplitude in counterfactual generation shifts the set of components identified as important, with critical regimes (e.g., noise amplitude $\approx$ 0.2) showing peak instability.

4. Non-Identifiability vs. Estimator Instability

The paper distinguishes between non-identifiability (a theoretical impossibility of recovering a unique circuit even with infinite data) and estimator instability (empirical sensitivity to finite sampling and approximation). While high instability is consistent with non-identifiability, the authors show that the observed variability is large enough to undermine current reporting practices regardless of the root cause.

Significance and Recommendations

The paper argues that for Mechanistic Interpretability to evolve into a rigorous science, it must adopt the standards of statistical inference used in natural sciences. The current practice of reporting single-point circuit discoveries without uncertainty estimates is insufficient.

Proposed Best Practices:

Report Stability Metrics: Researchers should routinely report stability metrics alongside circuit discovery results. Specifically, the variance of circuit structure (e.g., mean pairwise Jaccard index) and performance (CV of circuit error) under bootstrap resampling ( $n \ge 100$ $n \geq 100$ ).
- Suggested Threshold: A mean pairwise Jaccard index above 0.8 under bootstrap resampling could serve as a preliminary bar for reporting a circuit as stable.
Quantify Estimator Uncertainty: Sensitivity analyses regarding hyperparameters and aggregation methods should be conducted and reported. If a mechanism is only visible under specific hyperparameters, this fragility must be disclosed.
Characterize Intervention Sensitivity: Instead of relying on a single fixed intervention, researchers should analyze how circuits change as counterfactual definitions (e.g., noise amplitude) vary to distinguish core mechanisms from localized artifacts.

Conclusion:
The authors conclude that the field should move away from seeking a single "true" circuit. Instead, future work should aim to characterize a distribution over possible circuits (Probabilistic Circuit Discovery) and develop optimization objectives that explicitly penalize variance, prioritizing components that serve as reliable mediators across datasets and perturbations. This statistical framing is presented as a necessary step toward building trustworthy AI systems and ensuring interpretability tools provide a solid foundation for safety and policy decisions.