🤖 machine learning

From Mechanistic to Compositional Interpretability

This paper introduces "compositional interpretability," a category-theoretic framework that formalizes mechanistic explanations as commuting syntactic and semantic mappings to enable objective verification, optimization, and systematic model refinement toward more concise, human-aligned interpretations.

Original authors: Ward Gauderis, Thomas Dooms, Steven T. Holmer, Kola Ayonrinde, Geraint A. Wiggins

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ward Gauderis, Thomas Dooms, Steven T. Holmer, Kola Ayonrinde, Geraint A. Wiggins

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a incredibly complex, black-box machine that can do amazing things, like recognizing animals or translating languages. You know what it does, but you have no idea how it does it. Inside, it's a tangled mess of gears, wires, and levers that no human can easily understand. This is the current state of many advanced AI models.

The paper you provided proposes a new way to untangle this mess. It moves from just "guessing" how the machine works to creating a strict, mathematical rulebook for explaining it. Here is the core idea, broken down with simple analogies.

1. The Problem: The "Just-So" Story

Currently, when scientists try to explain these AI models, they often look at the inputs and outputs (e.g., "It saw a cat, so it said 'meow'"). They might point to a specific wire and say, "This wire lights up when it sees fur."

The paper argues this is risky. It's like looking at a car engine and saying, "This piston moves when the car goes fast." That might be true, but it doesn't explain the mechanism of how the fuel turns into motion. If you only look at the surface, you might tell a story that sounds right but is actually wrong about how the machine really works. This is called a "just-so story"—a tale that fits the facts but lacks the real internal logic.

2. The Solution: The "Commutative" Map

The authors introduce a concept called Compositional Interpretability. Think of it like a translation project for a complex machine.

To explain the machine properly, you need two maps that must perfectly match each other:

Map A (The Blueprint): This shows the machine's internal structure—the wires, the gears, the "syntax."
Map B (The Behavior): This shows what the machine actually does when you push a button—the "semantics."

The paper says a good explanation is only valid if these two maps are commutative. In plain English, this means:

If you follow the path of a specific gear in the Blueprint, it must lead to the exact same result as watching that gear move in the Behavior map.
If the maps don't line up, your explanation is broken. You can't just label a gear "Speed Controller" if it's actually doing something else.

This ensures that your explanation isn't just a guess; it's a verified link between the machine's guts and its actions.

3. The Goal: The "Shortest Story" (Occam's Razor)

Once you have a valid map, how do you know which explanation is the best? The paper uses a principle called Minimum Description Length.

Imagine you have to explain a complex magic trick to a friend.

Explanation 1: "The magician waved a wand, said a magic word, and the rabbit appeared." (Simple, but maybe misses the hidden trapdoor).
Explanation 2: "The magician used a hidden trapdoor, a spring mechanism, and a specific lighting angle to make the rabbit appear." (More complex, but accurate).
Explanation 3: "The magician used a hidden trapdoor, a spring, a lighting angle, a specific wind current, and a secret code." (Too complex, over-explaining).

The paper argues the best explanation is the shortest one that is still perfectly accurate. It's the sweet spot where you don't leave out important details (faithfulness) but you also don't add unnecessary fluff (complexity).

4. The Method: "Compressive Refinement"

How do we find this perfect, short explanation? The authors suggest a process called Compressive Refinement.

Think of the AI model as a messy pile of Lego bricks.

Current State: The bricks are glued together in weird, tangled clumps. It's hard to see what each piece does.
The Process: You carefully take the clumps apart and reassemble them into neat, distinct structures. You might add a few more "connectors" (wiring) to make the structure clear, but you simplify the individual pieces so they are easier to understand.
The Result: You end up with a model where the "atoms of computation" (the smallest useful parts) are simple and clear, and the way they connect is logical.

The paper proves that if you do this "re-assembly" correctly, you are guaranteed to get a simpler, more human-friendly explanation without changing what the machine actually does.

5. Why This Works: The "Optimism" Assumption

The paper makes a hopeful guess (a conjecture) to make this work: The patterns the AI uses to solve problems are likely the same patterns humans use to understand those problems.

If the AI is good at recognizing cats, it's likely using simple, logical features (ears, whiskers) rather than some alien, incomprehensible math. By compressing the model to find these simple patterns, we are essentially "filtering out" the noise and finding the human-readable logic hidden inside.

Summary

In short, this paper says:

Stop guessing: Don't just look at what the AI does; map its internal parts to its actions and make sure they match perfectly.
Keep it simple: The best explanation is the shortest one that is still 100% accurate.
Rearrange the machine: Systematically restructure the AI's internal wiring to make it simpler and clearer, which naturally leads to explanations humans can actually understand.

This provides a mathematical "rulebook" to turn the black box of AI into a transparent, understandable machine.

Technical Summary: From Mechanistic to Compositional Interpretability

Problem Statement
Mechanistic interpretability aims to reverse-engineer neural models into human-understandable components. However, current approaches lack a formal framework, rendering explanations difficult to objectively verify, compare, or compose. Existing methods often approximate faithfulness rather than enforcing it structurally. Furthermore, without a measurable definition of interpretation quality, explanations risk becoming "just-so stories" that correlate with surface-level behavior but fail to capture internal mechanisms. The core challenge is to define interpretability such that it is compositional (parts sum to the whole), faithful (matching internal computation), and concise (low description length for humans).

Methodology
The authors introduce Compositional Interpretability, a framework grounded in category theory and the Minimum Description Length (MDL) principle.

Formal Definitions via Category Theory:
- Compositional Model: Defined as a tuple $(D, S, C, [[\cdot]])$ , where $D$ is a string diagram representing the model's syntax (structure), $S$ is the syntactic category, $C$ is the semantic category (concrete implementation), and $[[\cdot]]$ is a functor mapping syntax to semantics.
- Compositional Interpretation: Defined as a pair of mappings: a syntax-based interpretation $I_S: S \to H$ (assigning explanations to structural components) and a semantics-based interpretation $I_C: C \to H$ (assigning explanations to behaviors).
- Commutativity: A valid interpretation requires the diagram to commute ( $I_S = I_C \circ [[\cdot]]$ ). This enforces consistency, ensuring that explanations derived from the model's structure align with explanations derived from its observed behavior.
Optimization via MDL:
- Interpretability is framed as a constrained optimization problem: finding the most concise explanation that remains faithful to the model's semantics.
- Faithfulness: Measured by the information cost required to correct an explanation to match the model's true semantics. The authors define compositional faithfulness, requiring that the explanation allows for the reconstruction of individual component semantics, not just global input-output behavior.
- Complexity: Measured by the Compositional Description Length ( $L(M, I_S)$ $L (M, I_{S})$ ), which decomposes into:
  - Representation Complexity ( $L_{rep}$ ): The cost of describing the syntax and the semantic complexity of atomic components.
  - Interpretation Complexity ( $L_{int}$ ): The subjective cost of aligning these components with human concepts.
- The total description length serves as an upper bound on the minimal description length of the explanation itself.
Compressive Refinement:
- To optimize interpretability, the paper proposes Syntactic Refinement: a functor $R: S \to S'$ that restructures the model's syntax while preserving semantics ( $[[\cdot]] = [[\cdot]]' \circ R$ ).
- Compressive Refinement occurs when this restructuring reduces representation complexity ( $L_{rep}(M') \leq L_{rep}(M)$ ).
- Parsimony Criterion: A refinement yields a better explanation only if the reduction in representation complexity outweighs any increase in interpretation complexity.

Key Contributions

Formal Framework: The paper establishes a category-theoretic foundation for interpretability, distinguishing between syntax (structure) and semantics (behavior) and requiring their alignment via commutative diagrams.
Quantifiable Objective: It casts interpretability as a constrained optimization problem using MDL, separating faithfulness (distortion) and complexity (description length).
Compressive Refinement: It introduces a systematic method to restructure models into simpler parts without altering function, generalizing existing mechanistic methods (like sparsity-based decomposition) as subclasses of refinement.
Parsimony Criterion & Theoretical Guarantees:
- The authors prove that compressive refinement guarantees more concise explanations if the coding distribution is comonotonic with the interpretation distribution (linking structural simplicity to human understandability).
- They invoke Compositional Sparsity (Theorem 4.5) to argue that deep networks of polynomial size approximate functions that admit sparse decompositions, ensuring the search for "atoms of computation" is feasible and does not lead to unmanageable diagram sizes.
- They formalize the Weak Principle of Explanatory Optimism, suggesting that if a model performs well, its core mechanisms are likely human-understandable, justifying the use of compression as a filter for these mechanisms.

Results and Claims
The paper does not present new empirical experiments but provides a theoretical unification and proof of concept:

It demonstrates that prominent mechanistic methods (e.g., those using sparsity heuristics) can be viewed as specific instances of syntactic refinement.
It proves that under the parsimony criterion, minimizing representation complexity theoretically guarantees more concise, human-aligned explanations, provided the "explanatory optimism" prior holds.
It clarifies why compressibility heuristics often align with human interpretability: they strip away redundancy to isolate core mechanisms that correspond to human concepts.

Significance
The paper claims to provide a measurable and optimizable foundation for automating the discovery and evaluation of mechanistic explanations. By moving from ad-hoc reverse-engineering to a formal compositional framework, it addresses the lack of objective verification in current interpretability research. The work situates interpretability as a continuous optimization objective rather than a binary architectural constraint, bridging the gap between intrinsic interpretability and post-hoc explanation. The authors suggest that future work can leverage this framework to integrate compressive refinement with causal inference and utilize tools like tensor networks for scalable optimization.