💬 NLP

Position: Mechanistic Interpretability Must Disclose Identification Assumptions for Causal Claims

This paper argues that mechanistic interpretability research frequently conflates validation metrics with causal identification by failing to explicitly state the necessary assumptions, and it proposes a new disclosure norm requiring authors to clearly articulate their identification strategies and the robustness of their causal claims.

Original authors: Zezheng Lin, Fengming Liu

Published 2026-05-11

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Zezheng Lin, Fengming Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to figure out why a complex machine (an AI) does what it does. You suspect a specific gear (a "circuit" or "feature") is the cause of a specific action. You pull that gear out, and the machine stops working. You declare, "Aha! That gear was the cause!"

This paper argues that in the field of AI "mechanistic interpretability," researchers are making these declarations far too often without following the rules of evidence. They are claiming to have found the cause, but they haven't explained why their experiment proves it's the cause and not just a coincidence.

Here is the breakdown of the paper's argument using simple analogies:

1. The Core Problem: "Validation" is not "Identification"

The paper makes a crucial distinction between two things:

Validation (The "What"): "I pulled the gear, and the machine stopped." This is a fact. It's a successful test.
Identification (The "Why"): "I know for a fact that only that gear caused the machine to stop, and not some hidden backup gear or a side effect of my pulling."

The Analogy:
Imagine you are a doctor. You give a patient a pill, and their fever goes down.

Validation: "The fever went down after the pill." (This is true).
Identification: "The pill caused the fever to go down." (This requires assumptions).

Maybe the fever went down because it was the time of day the fever naturally breaks. Maybe the patient drank water at the same time. Maybe the pill didn't work, but the fever was going away anyway.

In AI research, papers often say, "We changed this part of the AI, and the behavior changed, so this part is the cause." The paper argues they are skipping the step of proving that no other explanation could have caused that change. They are treating the "fever going down" (validation) as proof of the "pill working" (identification) without checking if the patient had a natural fever cycle.

2. The "Hidden Assumptions" Trap

Every time a researcher claims to have found a cause, they are standing on a set of invisible assumptions. The paper says these assumptions are currently hidden.

The Analogy:
Imagine a magician claims, "I made the rabbit disappear because I waved my wand."

The Hidden Assumption: "The rabbit didn't jump out the back door."
The Hidden Assumption: "The rabbit wasn't already gone before I waved."

In AI papers, the "magic tricks" are things like:

Activation Patching: "We swapped this part of the AI's brain. The behavior changed."
- Hidden Assumption: "We didn't accidentally wake up a sleeping backup system that also does this job."
Sparse Autoencoders (SAEs): "We found a specific 'feature' in the AI that represents the concept of 'honesty'."
- Hidden Assumption: "This feature is a unique, standalone building block, not just a messy mix of other things that happens to look like 'honesty'."

The paper audits 10 major papers (and checks 30 more) and finds that zero of them have a dedicated section where they list these hidden assumptions. They just show the magic trick (the result) and say "Look, it works!" without admitting what rules they are assuming the universe follows.

3. The "Substitution" Error

The paper calls the current habit "Validation Metric Substitution."

The Analogy:
Imagine a car mechanic says, "I fixed your engine because the car started."

The Error: The mechanic is substituting the result (the car started) for the proof (I know exactly which part I fixed and that nothing else could have started the car).
The Reality: The car might have started because the battery recharged itself, or the mechanic bumped the key, or the engine was fine all along.

In AI papers, researchers report metrics like "faithfulness" (does the explanation match the model?) or "completeness" (did we find all the parts?). The paper argues these are just "the car started" moments. They are good to have, but they are not proof of causality unless you state the assumptions that make them proof.

4. The Proposed Solution: The "Disclosure Protocol"

The author suggests that the field should borrow a rule from Econometrics (the study of economics data). In economics, if you claim "X causes Y," you must explicitly state:

What you are claiming: "We claim X causes Y."
The Strategy: "We are using method Z to prove it."
The Assumptions: "We assume that [Condition A] and [Condition B] are true."
The Stress Test: "If [Condition A] is false, our conclusion falls apart. Here is how we tested if it's true."

The Analogy:
Instead of just saying "The pill worked," the doctor must write a report:

"We claim the pill lowered the fever."
"We assume the patient didn't take other meds."
"We assume the fever wasn't naturally ending."
"If the patient did take other meds, our conclusion is wrong. Here is the data showing they didn't."

5. What the Audit Found

The author looked at 10 key papers in the field (covering different methods like circuit discovery and autoencoders) and then checked 30 more.

Result: 0 out of 30 papers had a dedicated section listing their identification assumptions.
Result: Most papers (about 19 to 26 out of 30, depending on how strict you are) used "validation metrics" (like "the AI behaved as expected") as a substitute for proving their assumptions were true.
Result: Even the most careful papers, which found errors in others' work, didn't explicitly list the assumptions they were using to find those errors.

Summary

The paper is a call for honesty and clarity. It doesn't say AI research is wrong; it says AI research is incomplete.

Right now, researchers are saying, "Look, we found the cause!"
The paper says, "Please stop and tell us what rules you are assuming to make that claim. If you don't tell us the rules, we don't know if your 'cause' is real or just a lucky guess."

It's asking the field to move from "Look how cool our magic trick is" to "Here is exactly how the trick works, and here is why we know it's not a trick."

Technical Summary: Mechanistic Interpretability Must Disclose Identification Assumptions for Causal Claims

Problem Statement
Mechanistic interpretability (MI) research increasingly employs causal vocabulary—such as "circuits," "mediators," "causal abstraction," and "monosemanticity"—to describe model behaviors. However, the paper argues that these causal claims frequently lack explicit identification assumptions. In causal inference, a claim is only valid if the conditions under which the causal quantity is identified from observable data are stated. The paper identifies a recurring pattern where researchers substitute validation metrics (e.g., faithfulness, completeness, monosemanticity, alignment, or ablation effects) for identification assumptions. While these metrics demonstrate that a description is consistent with the model or the data, they do not establish that the proposed mechanism is the cause rather than a correlate. The paper posits that without stating the assumptions under which a claim is identified, the causal claim remains unidentified, regardless of high validation scores.

Methodology
The paper employs a purposive audit of the MI literature to document the absence of identification assumption disclosure.

Primary Audit (n=10): A single-coder audit of ten influential papers spanning four methodological strands: circuit discovery, sparse autoencoders (SAE), causal abstraction, and probing. The sample includes works from NeurIPS, ICLR, CLeaR, BlackboxNLP, and the Anthropic Transformer Circuits Thread.
Extension Audit (n=30): A two-human-coder audit on an expanded set of 30 papers to test robustness and estimate inter-rater reliability.
Coding Dimensions:
- Dim A: Abstract claim type (explicit causal with/without ID basis, implicit causal, or association only).
- Dim B: Presence of a dedicated "Identification Assumptions" section in the main text.
- Dim C: Presence of falsifiability tests or sensitivity analyses.
- Dim D: "Validation-metric substitution" (reporting a validation metric as causal support without separating it from identification assumptions).
Validation: The study includes a second human-coder re-coding of a subset to measure agreement ( $\kappa$ ) and a single-rater audit on a stratified subsample to verify structural findings.

Key Contributions

Conceptual Distinction: The paper formally distinguishes between validation (answering "What changed when we intervened?") and identification (answering "Why does that change identify this mechanism?"). It argues that validation metrics are necessary but insufficient for causal claims.
Documentation of the Gap: The audit reveals a systemic lack of assumption disclosure. Across the sampled papers, no single paper contained a dedicated identification-assumptions section. The majority of papers implicitly made causal claims and substituted validation metrics for identification statements.
Method-Specific Assumption Enumeration: The paper enumerates the specific, often implicit, identification assumptions required for common MI methods:
- Activation Patching: Assumes circuit completeness (targeted components account for all pathways) and pathway exclusivity (patching does not activate dormant parallel circuits).
- Sparse Autoencoders: Assumes dictionary basis recoverability (features correspond to a true basis, not an arbitrary rotation) and feature atomicity/monosemanticity.
- Causal Abstraction: Assumes alignment faithfulness and optimization validity.
- Probing: Assumes causal locality and intervention completeness.
Proposed Disclosure Protocol: The paper proposes a five-step norm for MI papers making causal claims:
- State in the abstract whether the claim is causal or associative.
- Name the identification strategy.
- Include a numbered "Identification Assumptions" section in the main text.
- Provide a falsifiability test, sensitivity analysis, or clear counter-evidence statement for at least one core assumption.
- Discuss how conclusions shift if key assumptions fail.

Results

Absence of Dedicated Sections: In both the n=10 and n=30 audits, 0/30 papers contained a dedicated identification-assumptions section under any coding rule or coder.
Prevalence of Substitution: In the n=30 audit, 19/30 papers (under strict adjudication) or 26/30 (under permissive rules) were coded as substituting a validation metric for an identification statement.
Strand Heterogeneity: The substitution pattern was most concentrated in the Sparse Autoencoder and Causal Abstraction strands (6/6 and 3/3 papers, respectively, in the n=30 sample showed full substitution). Activation patching showed a lower rate of substitution, potentially due to existing literature on "illusion and failure modes" (e.g., Makelov et al., 2024) providing partial protection.
Inter-rater Reliability: Agreement on the presence of a dedicated section (Dim B) was low ( $\kappa = 0.26$ ), indicating that current papers do not provide stable, easily identifiable assumption statements. However, the structural finding of zero dedicated sections was robust across all coders and rules.

Significance and Claims
The paper claims that the current practice in mechanistic interpretability treats validation metrics as a proxy for causal identification, leading to "interpretability illusions" where high scores coexist with failed identification (as demonstrated by prior work like Makelov et al., 2024, and Leask et al., 2025).

Not a Framework Imposition: The paper does not import specific econometric frameworks (e.g., instrumental variables, do-calculus) into MI. Instead, it imports the meta-practice of causal inference: the requirement that causal claims be evaluated through stated assumptions.
Logical Necessity: The authors argue that disclosure is a logical requirement of any causal claim, not merely a stylistic choice. Without stated assumptions, the scientific community cannot evaluate whether a result reflects genuine causal structure or an artifact of unstated conditions.
Community Standard: The audit serves as evidence that the gap is widespread enough across major venues to warrant a community standard. The proposed protocol is designed to be generative rather than restrictive, enabling cumulative critique by making assumptions explicit and falsifiable.

The paper concludes that while the audit sample is small and purposive (not a prevalence estimate for the entire literature), the consistency of the pattern across strands and years suggests that the field has not yet adopted the baseline practice of assumption disclosure, despite the increasing reliance on causal language.