💬 NLP

Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable?

This paper investigates the identifiability of Mechanistic Interpretability by demonstrating through experiments on Boolean functions and small neural networks that unique explanations are often unattainable due to systematic non-identifiability, thereby prompting a re-evaluation of whether strict uniqueness is necessary for valid AI explanations.

Original authors: Maxime Méloux, Silviu Maniu, François Portet, Maxime Peyrard

Published 2026-06-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Maxime Méloux, Silviu Maniu, François Portet, Maxime Peyrard

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, black-box robot that can solve a specific puzzle perfectly. You want to know how it does it. You don't just want to know that it works; you want to pull back the curtain and see the tiny gears, levers, and switches inside that make the magic happen. This is the goal of Mechanistic Interpretability: trying to reverse-engineer a neural network (the robot's brain) to find a simple, human-readable story about how it thinks.

This paper asks a very simple, yet profound question: If we look inside this robot to find the story of how it works, will we all find the exact same story? Or, could two different people look at the same robot, use the same rules to investigate, and come up with two completely different, yet equally valid, stories?

The authors say: No, we are not guaranteed a unique story. In fact, there are often dozens, hundreds, or even thousands of different "stories" that fit the data perfectly.

Here is a breakdown of their findings using simple analogies:

The Two Ways to Investigate

The paper looks at two main ways researchers try to find these stories:

"Where-then-What" (The Detective Approach):
- The Idea: First, you try to find a small group of neurons (a "circuit") inside the robot that does all the heavy lifting. Once you find that group, you try to guess what logic they are using (e.g., "Oh, these three neurons act like an AND gate").
- The Problem: Imagine you have a giant library of books. You want to find the specific pages that tell a story. You might find one set of pages that tells the story perfectly. But then you find another set of pages, completely different from the first, that also tells the story perfectly. The paper found that for a simple task, there were 85 different groups of neurons that could all do the job perfectly. There is no single "correct" group of gears.
"What-then-Where" (The Architect Approach):
- The Idea: First, you guess a simple story or algorithm (e.g., "The robot must be using an OR gate"). Then, you scan the robot's brain to see if you can find a place where that story is being played out.
- The Problem: Imagine you are looking for a specific melody being played in a massive orchestra. You might find the melody being played by the violins. But then you realize the cellos are playing the exact same melody at the same time, and the flutes are doing it too. The paper found that for a single simple algorithm, there were 159 different places in the robot's brain where that algorithm was being implemented perfectly.

The "XOR" Experiment

To prove this, the authors trained a tiny, simple robot to solve a classic logic puzzle called XOR (which is like a light switch that turns on only if one of two buttons is pressed, but not both).

The Result: They didn't just find one explanation. They found over 45,000 different explanations (combinations of circuits and stories) that were all mathematically perfect.
The Analogy: It's like asking, "How did the baker make this perfect cake?"
- Explanation A: "She used flour, sugar, and eggs in Bowl X."
- Explanation B: "She used flour, sugar, and eggs in Bowl Y."
- Explanation C: "She used a different mix of ingredients in Bowl Z."
- All of them are true. The cake tastes the same, but the "mechanism" you point to is different every time.

Why Does This Matter?

In statistics, we have a concept called identifiability. It means that if you have the data, there should be only one set of "true" numbers that created it. This paper argues that for AI explanations, identifiability is broken.

The "Unique Truth" Myth: We often assume that if we dig deep enough, we will find the one true way the AI thinks. This paper says that assumption is likely wrong. The AI might be thinking in many different ways simultaneously, or there might be many different ways to describe its thinking that are all equally correct.
The Consequence: If you ask two experts to explain the same AI, they might give you two totally different answers, and both would be right according to the current rules of science.

The Conclusion: What Should We Do?

The authors don't say "give up." Instead, they suggest we need to change our mindset:

Stop looking for the "One True Story": We might need to accept that there isn't a single, unique explanation.
Focus on Utility: Maybe it doesn't matter if the explanation is unique. Maybe it only matters if the explanation helps us predict what the AI will do next or manipulate it to do what we want.
Use Multiple Checks: If we really need to be sure, we shouldn't rely on just one method. We should use many different tests to see if an explanation holds up from every angle.

In short: The paper argues that the "fingerprint" of an AI's thinking is not unique. There are many different keys that can open the same lock, and many different maps that can describe the same territory. We need to stop expecting a single, perfect map and start accepting that the "truth" of how AI works might be a collection of many valid, overlapping stories.

Technical Summary: "Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable?"

Problem Statement

As AI systems are deployed in high-stakes applications, ensuring their interpretability is critical. Mechanistic Interpretability (MI) aims to reverse-engineer neural networks by extracting human-understandable algorithms (computational abstractions) embedded within their structures. A computational abstraction consists of two components: the what (the explanatory algorithm) and the where (the specific subgraph or circuit within the network where this algorithm is implemented).

While MI methods often operate under a tacit assumption of explanatory unicity—the belief that a unique, correct explanation exists for a model's behavior—this paper systematically questions this premise. Drawing an analogy to identifiability in statistics (where parameters are uniquely inferred from data under specific assumptions), the authors ask: Under the criteria set by MI, is the explanation for a fixed behavior guaranteed to be unique?

The paper investigates two primary strategies for generating MI explanations:

Where-then-what: Detecting a subset of the network (a circuit) that replicates the model's behavior, then deriving the interpretation of its components.
What-then-where: Starting with candidate explanatory algorithms and searching for activation subspaces in the network where these algorithms are causally aligned.

The core problem is whether these strategies, under their respective validity criteria (circuit error and causal alignment), yield a single, unique explanation or if multiple, incompatible explanations can satisfy the same criteria.

Methodology

The authors conduct a stress-test of MI identifiability using a controlled, small-scale setting that allows for exhaustive enumeration of candidate explanations, a feat impossible in large-scale models.

Experimental Setup

Tasks: Training Multi-Layer Perceptrons (MLPs) to implement Boolean functions, specifically the XOR function and combinations of logic gates.
Architecture: Small MLPs (e.g., input size 2, hidden layers of size $k$ , output size $n$ ) trained on binary inputs with added Gaussian noise until a low loss threshold is reached.
Exhaustive Search:
- For "Where-then-what": The authors enumerate all possible subgraphs (circuits) of the network. They test each circuit on validation data to identify "perfect circuits" (zero circuit error). For each perfect circuit, they recursively search for all valid mappings (interpretations) of neuron activations to logic gates (AND, OR, NOT) that satisfy consistency (the mapping commutes with the computation).
- For "What-then-where": The authors enumerate candidate Boolean algorithms (parse trees) implementing the target function. They then search for all possible subsets of neurons and mappings that achieve perfect causal alignment with the algorithm, measured by Intervention Interchange Accuracy (IIA). An IIA of 1 indicates that counterfactual interventions on the algorithm produce identical effects on the network's internal states.

Definitions of Identifiability

The paper defines an explanation as non-identifiable if the current MI criteria fail to discriminate between two incompatible explanations. Incompatibility arises if:

Two explanations posit different algorithms for the same behavior.
The same algorithm is embedded in different subspaces of the network.

Key Results

The experiments reveal overwhelming evidence of non-identifiability across all stages of the MI process:

Non-Unique Circuits (Where): Multiple distinct circuits can perfectly replicate the model's behavior (zero circuit error). In the XOR example, 85 unique circuits with perfect accuracy were found.
Non-Unique Interpretations (What): For a single valid circuit, multiple valid interpretations (mappings of neurons to logic gates) exist. In the XOR example, an average of 535.8 interpretations were found per circuit.
Non-Unique Algorithms (What): Several different algorithms can be perfectly causally aligned with the neural computation (IIA = 1). Even when testing only two candidate algorithms for XOR, both were found to have perfect implementations.
Non-Unique Localizations (Where): For a single causally aligned algorithm, multiple distinct subspaces of the network can be equally aligned (IIA = 1). In the XOR example, 159 perfect minimal mappings were identified for just two candidate algorithms.

Quantitative Findings:

As network size ( $k$ ) increases, the number of valid computational abstractions grows exponentially (median increasing from 38 to 910,000 in the circuit-first approach).
Less than 2% of trained networks contained exactly one valid minimal mapping, and no network contained exactly one circuit interpretation.
Increasing task complexity (training on multiple logic gates simultaneously) reduced the number of interpretations but did not eliminate non-identifiability.

Key Contributions

Formalization of Identifiability in MI: The paper introduces the concept of "identifiability of explanation" to the field of mechanistic interpretability, adapting a statistical concept to define the uniqueness of explanations under fixed validity criteria.
Systematic Stress-Testing: By utilizing small-scale models to allow exhaustive enumeration, the authors provide the first rigorous, empirical demonstration that current MI criteria (circuit error and IIA) do not guarantee unique explanations.
Categorization of Strategies: The work clearly delineates and evaluates the "where-then-what" and "what-then-where" strategies, showing that both suffer from non-identifiability issues.
Empirical Evidence of Incompatibility: The study demonstrates that a single trained model can support hundreds of thousands of mutually incompatible computational abstractions that all satisfy the standard criteria for a "good" explanation.

Significance and Claims

The authors state that this work aims to contribute constructively to the ongoing effort to formalize expectations for AI explanations. The paper does not claim that MI is useless, but rather challenges the "unicity intuition" often assumed in the field.

Pragmatic Stance: The authors suggest that if the goal is prediction or manipulability, unicity may not be necessary, and one could adopt a pragmatic stance accepting multiple valid explanations.
Strict Stance: If unicity is considered essential for providing a genuine "sense of understanding," the paper argues that current criteria are insufficient and less permissive criteria (or the "inner interpretability" framework requiring multiple complementary validations) may be required.
Future Direction: The work highlights the need for rigorous definitions of what constitutes an explanation in AI, moving beyond the assumption that a single "true" mechanism exists and is easily discoverable via current methods.

The paper concludes that the current landscape of Mechanistic Interpretability faces a fundamental identifiability problem: without additional constraints or criteria, we cannot guarantee that the explanations we derive are the unique, correct mechanisms governing a model's behavior.