🤖 machine learning

Mechanistic Interpretability for Neural Networks: Circuits, Sparse Features and Symbolic Reasoning

This paper provides a comprehensive overview of mechanistic interpretability, detailing how techniques like Transformer circuit analysis, Sparse Autoencoders, and causal interventions can reverse-engineer neural networks to uncover internal algorithms, resolve polysemanticity, and translate representations into explicit logical rules for safer AI.

Original authors: Pranav Sawant, Jakub Krejčí

Published 2026-07-09

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Pranav Sawant, Jakub Krejčí

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Opening the Black Box

Imagine modern AI models (like the ones that write essays or chat with you) as giant, sealed black boxes. We know what goes in (a question) and what comes out (an answer), but we have no idea what happens inside the millions of gears and wires in between.

Traditional ways of trying to understand these boxes are like looking at a car's exhaust pipe to guess how the engine works. They tell you what the car is doing, but not how the engine is making it happen.

This paper introduces a new approach called Mechanistic Interpretability. Instead of just guessing, this field tries to reverse-engineer the AI. It's like taking the black box apart, laying out every single gear, spring, and wire, and writing down the exact instructions (pseudocode) for how they work together to produce a result.

1. The Highway and the Traffic (Circuits)

Inside these AI models, there is a central "highway" called the residual stream. Think of this as a conveyor belt carrying information from the start of the model to the end.

The Traffic Controllers (Attention Heads): These are like traffic cops on the conveyor belt. They decide which piece of information needs to look at which other piece. For example, if the sentence is "The cat sat on the mat," a traffic cop might link the word "it" back to "cat."
The Workers (MLP Layers): These are like workers who take the information on the belt and transform it. They add new ideas or change the meaning.
The Magic Trick (Induction Heads): The paper highlights a specific type of traffic cop called an "induction head." Imagine you are reading a story where a pattern repeats: "Aunt Sally... Aunt Sally." An induction head is the part of the brain that notices, "Hey, I've seen this pattern before! The next word is probably 'Sally' again." This is how the AI learns new things just by reading a prompt, without needing to be retrained.

2. The Problem: The "Superposition" Soup

Here is the tricky part. If you look at a single neuron (a tiny processing unit) in the AI, it doesn't usually do just one thing. It's like a Swiss Army Knife that is trying to be a screwdriver, a bottle opener, and a corkscrew all at the same time.

Polysemanticity: A single neuron might fire when it sees a picture of a cat, a mention of "Michael Jordan," or the word "basketball." It's doing too many jobs at once.
Superposition: The AI is so efficient that it crams thousands of different ideas into a limited number of neurons. It's like trying to fit 100 different flavors of ice cream into a single cup; the flavors get mixed up.

This makes it very hard to understand the AI because you can't just point to one neuron and say, "This is the 'cat' neuron."

3. The Solution: The "Feature Blender" (Sparse Autoencoders)

To fix the "Swiss Army Knife" problem, the paper discusses a tool called a Sparse Autoencoder (SAE).

Think of the AI's messy, mixed-up thoughts as a smoothie where all the fruits are blended together. You can't taste the strawberry or the banana separately.
The SAE is a machine that takes that smoothie and de-blends it. It separates the mixture back into distinct, pure ingredients.

Instead of a neuron that is half "cat" and half "basketball," the SAE finds two separate "pure" features: one that is 100% "cat" and one that is 100% "basketball."
This allows researchers to see exactly what the AI is thinking about at any given moment.

4. Finding the Hidden Circuits (Automated Discovery)

Manually tracing every wire in the AI is impossible because there are too many. So, researchers use automated tools like ACDC (Automated Circuit Discovery).

Imagine the AI is a giant, tangled ball of Christmas lights. You want to find the specific string of lights that makes the tree turn on.

ACDC acts like a robot that systematically cuts one wire at a time.
If cutting a wire doesn't change the outcome, it's a useless wire, so it's removed.
If cutting a wire stops the tree from turning on, that wire is part of the "circuit."
By the end, the robot has stripped away all the junk and left you with a clean, simple diagram of exactly which wires are needed to do the job.

5. Steering the AI (The Volume Knob)

Once we understand the circuits, we can try to control them. The paper talks about Steering Vectors.

Think of the AI's internal state as a radio. Usually, you have to shout instructions (like typing "Be polite") to change the station.

Steering Vectors are like a remote control that whispers directly into the radio's internal wiring.
Instead of shouting, you just nudge the internal signal slightly in a specific direction.
For example, if you nudge the signal toward "politeness," the AI becomes polite without you having to rewrite its entire personality.
The paper notes this can be used to stop the AI from lying (by amplifying an "honesty" feature) or to make it solve math problems better.

6. Translating to Human Logic (Neurosymbolic AI)

Finally, the paper discusses Neurosymbolic AI. This is like taking the complex, messy code of the AI and translating it into a simple instruction manual that a human can read.

Imagine the AI is a wizard casting a complex spell.
Neurosymbolic frameworks take that spell and write it down as a simple "If-Then" rule: "If the image has a bed and a pillow, but no sink, then it is a bedroom."
This turns the "black box" into a transparent set of logical rules that we can check, verify, and trust.

Summary

The paper argues that we are moving past the era of just guessing how AI works. By using tools to de-blend mixed-up ideas (SAEs), trace the exact paths of information (Circuits), and nudge the internal signals (Steering), we are starting to understand the internal "gears" of these machines. This is crucial for making sure these powerful tools are safe, honest, and do exactly what we want them to do.

Technical Summary: Mechanistic Interpretability for Neural Networks

Problem Statement

Modern Transformer-based Large Language Models (LLMs) and Vision Transformers (ViTs) function as complex "black boxes." While traditional explainable AI (XAI) methods, such as saliency maps, LIME, and SHAP, provide surface-level correlations between inputs and outputs, they fail to reverse-engineer the internal algorithms responsible for model behavior. This opacity poses significant risks in high-stakes domains (e.g., healthcare, law, autonomous driving) where safety, auditability, and trust are paramount.

A core obstacle to understanding these models is superposition and polysemanticity. Neural networks compress a vast number of features ( $M$ ) into a limited number of dimensions ( $D$ , where $M \gg D$ ). Consequently, individual neurons often respond to mixtures of unrelated concepts (polysemanticity) rather than single, well-defined features (monosemanticity). This distribution of meaning across high-dimensional spaces renders traditional neuron-level analysis misleading. Furthermore, while attention visualizations reveal where information is routed, they do not always explain what information is represented or whether it is causally necessary for the output.

Methodology

This paper presents a scoping review of Mechanistic Interpretability (MI), a field dedicated to reverse-engineering neural networks into human-interpretable algorithms, pseudocode, and functional components. The methodology follows the PRISMA-ScR guidelines, mapping the landscape of MI tools and concepts without mandatory critical appraisal of individual study quality.

The review focuses on three primary methodological pillars:

Circuit Analysis: Decomposing models into causal subnetworks (circuits) composed of attention heads, MLP layers, and residual streams.
Feature Disentanglement: Using Sparse Autoencoders (SAEs) and transcoders to separate polysemantic activations into monosemantic features.
Intervention and Neurosymbolic Integration: Modifying model behavior via steering vectors and translating neural representations into explicit logical rules.

The review prioritizes research from 2018 to 2026, with a specific emphasis on Transformer architectures (LLMs, ViTs, VLA) and methods such as Automated Circuit Discovery (ACDC), Activation Patching, and Linear Artificial Tomography (LAT).

Key Contributions and Technical Findings

1. Transformer Circuit Analysis

The paper details how Transformer models can be reverse-engineered by analyzing the residual stream (the central communication channel) and attention mechanisms.

Circuit Decomposition: The review highlights the Universality Hypothesis, which posits that similar models trained on similar data develop similar internal circuits. This allows for the identification of reusable computational paths.
Induction Heads: A critical finding is the identification of induction heads, which are responsible for in-context learning. These heads operate via a three-part sub-circuit: (A) attending to the previous token, (B) matching the query-key to find a pattern, and (C) copying the subsequent token. This mechanism emerges via a sharp phase transition and is additive; the network possesses massive redundancy where weaker heads can compensate if stronger ones are ablated.
Indirect Object Identification (IOI): The IOI task serves as a benchmark for circuit analysis. In GPT-2 Small, the task is solved by a specific circuit of 26 attention heads (Duplicate Token Heads, S-Inhibition Heads, and Name Mover Heads). The paper notes that this circuit structure varies in non-Transformer architectures (e.g., Mamba), demonstrating that interpretation tools must be architecture-aware.
Automated Discovery: To address the time-consumption of manual analysis, the paper reviews Automated Circuit Discovery (ACDC), which uses activation patching to prune irrelevant edges in a computational graph. More scalable methods like Edge Attribution Patching (EAP) and ALTI use gradient linearization to attribute importance to connections in a single forward/backward pass, eliminating the need for artificial contrastive templates.

2. Superposition and Sparse Autoencoders (SAEs)

To address polysemanticity, the paper focuses on Sparse Autoencoders (SAEs).

Mechanism: SAEs are trained on the activations of a frozen base model to decompose dense, polysemantic vectors into a wide, overcomplete dictionary of sparse, monosemantic features.
Architectures: The review compares standard SAEs with advanced variants like Gated SAEs, TopK SAEs, and JumpReLU SAEs, which mitigate issues like the "shrinkage effect" and dead latent features.
Scaling Laws: Reconstruction error decreases as a power function with the number of latent features, suggesting that fully disentangling frontier models may require dictionaries with tens to hundreds of millions of features.
Transcoders: While SAEs reconstruct static activations, Transcoders (and Cross-Layer Transcoders) learn to mimic the input-output behavior of entire MLP layers or multi-layer transformations. This allows researchers to trace how information is transformed rather than just what is represented.
Multilingual Processing: Using Cross-Layer Transcoders, the paper reveals that multilingual models process text in three phases: language-specific encoding, a shared language-agnostic semantic space (middle layers), and language-specific decoding. This shared space is an architectural property, not merely an artifact of English dominance in training data.

3. Steering Vectors and Activation Interventions

The paper explores steering vectors as a method for causal intervention.

Linear Representation Hypothesis (LRH): This hypothesis suggests that high-level concepts (e.g., truthfulness, sentiment) are encoded as linear directions in activation space.
Extraction Methods: Techniques range from simple activation differences (ActAdd) to robust contrastive methods (CAA) and Linear Artificial Tomography (LAT).
SAE-Based Steering: By using SAEs to isolate specific features, researchers can perform feature clamping (fixing or amplifying a feature's activation) to causally alter model behavior. For instance, amplifying "desperation" vectors can induce harmful strategies, while amplifying "internal conflict" vectors can force a model to reveal hidden information.
Dynamic Control: Methods like Dynamic Sparse Prompt-conditional Activation steering (DSPA) allow for context-sensitive steering by analyzing prompts in early layers and activating control features in later layers.

4. Neurosymbolic Frameworks

The paper connects MI with Neurosymbolic AI (NeSy), which aims to extract explicit logical rules from neural networks.

Rule Extraction: Frameworks like NeSyFOLD (for CNNs) and NeSyViT (for Vision Transformers) extract stratified logical rules (Answer Set Programs) from model activations. These rules use default logic (e.g., "Birds fly" with exceptions "unless penguins") to approximate decision boundaries.
Integration: Unlike pure extraction, frameworks like Logic-LM and LLM-Modulo integrate LLMs with external symbolic solvers. The LLM translates natural language into formal logic, which is then solved by a deterministic solver, providing correctness guarantees that pure neural reasoning lacks.

Results and Claims of Significance

The paper claims that mechanistic interpretability has successfully moved beyond surface-level correlations to identify causal, algorithmic accounts of neural network behavior.

Validation of Structure: The field has demonstrated that complex behaviors (e.g., in-context learning, indirect object identification) are not arbitrary but are implemented by structured, identifiable circuits.
Causal Control: The ability to steer models via specific features (e.g., suppressing "dangerous code" features or amplifying "truthfulness") provides empirical evidence that internal representations are causally linked to output behavior.
Safety Implications: The paper argues that understanding internal mechanisms is essential for AI safety. Conventional alignment (e.g., RLHF) may only mask undesirable behaviors at the surface level, whereas mechanistic interpretability can detect and modify the underlying reasoning strategies.
Limitations: The authors modestly note that most detailed circuit analyses are currently limited to small models or synthetic tasks. There is a significant gap between explaining GPT-2 Small and explaining reasoning or deception in frontier-scale systems. Additionally, while SAEs improve interpretability, they are not a complete solution, as they may fragment concepts or fail to capture dynamic transformations without transcoders.

Conclusion

The paper concludes that mechanistic interpretability is a necessary evolution for ensuring the safety and auditability of AI. While the field has made substantial progress in reverse-engineering specific circuits and features, the central open question remains whether these techniques can scale to the complexity of general-purpose frontier models. Future work must focus on scalable automated discovery, better evaluation benchmarks, and the development of models that are "interpretable by design."