Technical Summary: Language Model Circuits Are Sparse in the Neuron Basis

Problem Statement

Language model interpretability research has largely moved away from analyzing individual neurons, operating under the belief that high-level concepts are distributed across neurons and that the neuron basis is uninterpretable. Consequently, the field has adopted techniques like Sparse Autoencoders (SAEs) and transcoders to decompose representations into sparse, interpretable feature bases. However, these learned bases introduce approximation errors, suffer from feature splitting and absorption (polysemanticity), and require additional training overhead. The central question this paper addresses is whether the original neuron basis is inherently uninterpretable, or if the perceived lack of sparsity is an artifact of suboptimal analysis methods.

Methodology

The authors propose an end-to-end gradient-based attribution pipeline to trace circuits directly on the MLP neuron basis (specifically pre-down projection activations) rather than learned SAE features. The methodology relies on two key technical shifts to close the performance gap between neurons and SAEs:

1. Privileged Basis Selection: Instead of using post-down projection MLP outputs (the standard baseline), the authors utilize pre-down projection MLP activations. They hypothesize that because the architecture applies element-wise non-linearities (e.g., SiLU) before the down-projection, these activation coordinates constitute a "privileged basis" where features are more naturally aligned, yielding significantly sparser circuits.
2. Advanced Attribution (RelP): The paper replaces Integrated Gradients (IG), which is noisy on deep models and computationally expensive (requiring multiple backward passes), with RelP (Relevance Patching). RelP is a gradient-based attribution method that operates on a "replacement model" where non-linearities are linearized (e.g., treating SiLU gates as constants). This allows for faithful circuit discovery with a single backward pass. The authors implement a "half-rule" for gated MLPs to ensure the completeness property of attribution scores is preserved.

The pipeline evaluates circuits using two standard metrics:

• Faithfulness: The ability of the circuit's complement (the rest of the model) to reproduce the original model's behavior when the circuit is ablated.
• Completeness: The ability of the circuit itself to reproduce the model's behavior when the rest of the model is ablated.

Experiments were conducted on the Subject-Verb Agreement (SVA) benchmark (Marks et al., 2025) and a multi-hop city-state-capital reasoning task (Lindsey et al., 2025) using Llama 3.1-8B-Instruct, with additional validation on Gemma-2 models.

Key Results

1. MLP Activations are a Sparse Basis

The authors demonstrate that MLP activations yield circuits that are orders of magnitude sparser than those derived from MLP outputs.

• On the SVA benchmark, circuits constructed from MLP activations are approximately 100x smaller than those from MLP outputs while achieving comparable faithfulness and completeness.
• This finding significantly narrows the gap between the neuron basis and SAE-based representations.

2. RelP Outperforms Integrated Gradients

Switching from IG to RelP further improves sparsity and faithfulness.

• Using RelP on MLP activations, the authors achieve near-perfect faithfulness and completeness with only ~200 neurons on the SVA benchmark.
• RelP closes the remaining gap between neuron-based circuits and SAE-based circuits, making neurons competitive with learned dictionaries in terms of sparsity.
• On the Mechanistic Interpretability Benchmark (MIB), RelP achieves state-of-the-art performance (lowest CMD scores) on three out of five tasks, outperforming existing IG-based baselines.

3. Case Study: Multi-Hop Reasoning

In a case study involving the retrieval of state capitals (e.g., "What is the capital of the state containing Dallas?"), the authors traced circuits in Llama 3.1-8B-Instruct.

• They identified a sparse circuit of ~23 key neurons (out of a larger traced set) that correspond to specific reasoning steps: mapping a city to a state, and mapping a state to its capital.
• These neurons clustered into groups with interpretable descriptions (e.g., "Dallas," "Texas," "capital") that matched findings from prior work using Cross-Layer Transcoders (CLTs).
• Steering Experiments: By intervening on specific neuron clusters (e.g., suppressing the "Texas" cluster), the authors could causally alter the model's output, confirming that these specific neurons are causally effective in the reasoning chain.

4. Generalizability

The findings hold across different model architectures (Gemma-2 2B and 9B) and different data settings (paired counterfactuals and unpaired data with zero baselines).

Significance and Claims

The paper claims to provide the first empirical evidence that MLP neurons are as sparse a feature basis as SAEs, challenging the prevailing assumption that neurons are uninterpretable.

• Practicality: The neuron basis offers a "privileged" and faithful representation without the reconstruction errors, splitting, or absorption issues inherent in learned dictionaries.
• Efficiency: The proposed pipeline avoids the computational cost of training SAEs or transcoders, requiring only a single backward pass per input via RelP.
• Reproducibility: The ability to reproduce complex circuit findings (such as those from Lindsey et al., 2025) using only the native neuron basis suggests that the "dark matter" of language models may be more accessible than previously thought.

The authors conclude that while human interpretation of large circuits remains a challenge, the neuron basis should be exhausted as a primary avenue for interpretability before resorting to learned sparse dictionaries. They frame this work as a step toward democratizing AI safety research by providing efficient, scalable tools for tracing internal computations.