💬 NLP

Disentangling MLP Neuron Weights in Vocabulary Space

This paper introduces ROTATE, a data-free method that disentangles MLP neurons into interpretable "vocabulary channels" by optimizing weight rotations to maximize vocabulary-space kurtosis, thereby recovering sparse, faithful representations that outperform activation-based baselines in describing neuron behavior.

Original authors: Asaf Avrahamy, Yoav Gur-Arieh, Mor Geva

Published 2026-04-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Asaf Avrahamy, Yoav Gur-Arieh, Mor Geva

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a large language model (like the ones powering chatbots) as a massive, bustling city. Inside this city, there are millions of tiny workers called neurons. Each neuron is responsible for processing information, but they are notoriously messy.

Usually, one neuron is like a "Swiss Army Knife" worker: it handles math, it handles emotions, it handles time, and it handles coding all at the same time. If you ask, "What does this worker do?" the answer is a confusing jumble of everything. This is called polysemanticity, and it makes it very hard to understand how the AI actually thinks.

The paper you shared introduces a new tool called ROTATE to solve this mess. Here is how it works, explained simply:

1. The Problem: The "Swiss Army Knife" Neuron

Think of a neuron's "brain" as a giant, tangled ball of yarn. Inside that ball, there are many different strings, each representing a different concept (like "time," "negation," or "coding"). Right now, these strings are knotted together. If you pull on the ball, you can't tell which string is which.

2. The Solution: The "Magic Unknotter" (ROTATE)

The authors created a method called ROTATE. Instead of asking the AI to read millions of books to figure out what a neuron does (which is slow and expensive), ROTATE looks directly at the weights (the mathematical settings) of the neuron.

They treat the neuron's weight vector like a piece of clay. They use a mathematical "rotation" to spin this clay around until they find the angles where the clay looks the most "spiky" or "concentrated."

The Analogy of the "Spiky Distribution":
Imagine you have a bag of marbles.

A messy neuron is like a bag where the marbles are spread out evenly. It's boring and tells you nothing specific.
A "monosemantic" (single-meaning) neuron is like a bag where 99% of the marbles are in one corner, and only a few are scattered elsewhere. This is a "spiky" distribution.

ROTATE spins the clay until it finds these "spiky" corners. When it finds a spike, it knows it has isolated a single, clear concept.

3. The Discovery: "Vocabulary Channels"

Once ROTATE finds these spikes, it cuts the tangled ball of yarn into separate, clean strings. They call these Vocabulary Channels.

Before ROTATE: You have one neuron that vaguely "knows" about time and math.
After ROTATE: You have three distinct channels:
1. Channel A: Only talks about "Time" (words like until, wait, years).
2. Channel B: Only talks about "Negation" (words like not, didn't, never).
3. Channel C: Only talks about "Math" (words like plus, minus, equals).

4. Why This is a Big Deal

The paper proves two amazing things:

It's Data-Free: You don't need to feed the AI millions of sentences to figure this out. You just look at the math inside the model's brain. It's like figuring out what a tool does just by looking at its shape, without ever using it.
It's More Accurate: When the researchers tested this against other methods (like trying to guess what a neuron does by watching it work), ROTATE was 2 to 3 times better. It didn't just guess; it found the exact "switch" for specific concepts.

5. The "Causal" Test

To prove they really found the right strings, they did a "surgery." They took the "Time" channel and surgically removed it from the neuron.

Result: The neuron stopped talking about time completely, but it kept talking about math and negation perfectly fine.
Conclusion: They didn't just guess; they actually isolated the specific part of the brain responsible for that concept.

Summary

Imagine you have a chaotic orchestra where every musician is playing three different songs at once. ROTATE is a new conductor who can instantly separate the musicians into three distinct groups: the violins, the drums, and the flutes.

By doing this without needing to listen to the whole concert first, ROTATE gives us a clear, fine-grained map of how AI models think, making them less of a "black box" and more of a transparent machine we can understand.

1. Problem Statement

Mechanistic interpretability aims to understand how Large Language Models (LLMs) process information. A significant challenge is polysemanticity, where individual neurons (specifically in Multi-Layer Perceptron, or MLP, layers) encode multiple, entangled concepts simultaneously. This makes it difficult to attribute specific behaviors to specific neurons.

Existing approaches often rely on Sparse Autoencoders (SAEs) trained on activation data (forward passes) to decompose neurons into monosemantic features. However, these methods have limitations:

Data Dependency: They require massive datasets of activation vectors, which is computationally expensive and introduces biases based on the specific data distribution used.
Indirect Analysis: They analyze the residual stream activations rather than the weights themselves, potentially missing information encoded directly in the model parameters.
Faithfulness: SAE features may not always perfectly align with the causal behavior of the original neuron.

The authors ask: Can we disentangle MLP neuron weights directly in weight space without any forward passes or training data?

2. Methodology: ROTATE

The paper introduces ROTATE (Rotation-Optimized Token Alignment in TweighT spacE), a data-free method that decomposes a neuron's weight vector into sparse, interpretable "vocabulary channels."

Core Hypothesis

The authors observe that neurons encoding a single, coherent (monosemantic) concept exhibit high kurtosis when their weight vectors are projected into the model's vocabulary space.

Vocabulary Projection: A weight vector $w$ is projected via the unembedding matrix $U$ to produce logits $z = wU$.
Kurtosis: A statistical measure of the "tailedness" of a distribution. High kurtosis indicates a distribution that is sharply peaked with heavy tails (i.e., the neuron strongly promotes or suppresses a very small set of tokens while having little effect on the rest).
Insight: Low-kurtosis (polysemantic) neurons are likely composed of multiple high-kurtosis (monosemantic) directions. Maximizing kurtosis can isolate these directions.

Algorithm

ROTATE optimizes a rotation matrix $R$ to transform a neuron's weight vector $w$ into a new vector $v = wR$ such that the vocabulary projection of $v$ has maximum kurtosis.

Optimization Objective: The method minimizes a loss function $L$ composed of two terms:
- Kurtosis Loss ( $L_{kurt}$ ): Maximizes the kurtosis of the projected logits $z = vU$.
- Regularization Loss ( $L_{reg}$ ): Penalizes the cosine distance between the rotated vector $v$ and the original weight $w$ . This ensures the discovered channel remains a valid decomposition of the original neuron rather than converging to arbitrary high-kurtosis noise.
  $L = -\lambda \cdot \log(1 + \text{Kurt}(z)) + \left(1 - \frac{w \cdot v}{\|w\|\|v\|}\right)$
Parameterization: The rotation matrix $R$ is parameterized using a Householder reflection ( $R = I - 2\frac{hh^T}{\|h\|^2}$ ), which enforces orthogonality by construction.
Iterative Disentanglement: Since a single rotation yields only one channel, ROTATE runs iteratively. After finding a channel, it masks the tokens that contributed most to that channel's kurtosis. This forces subsequent iterations to discover new, distinct high-kurtosis directions, preventing the algorithm from converging to the same local optimum repeatedly.
Glitch Token Handling: The method explicitly masks known "glitch tokens" (under-trained embeddings with extreme norms) to prevent the optimizer from exploiting tokenizer artifacts.

3. Key Contributions

Data-Free Decomposition: ROTATE is the first method to disentangle MLP weights entirely in weight space without requiring forward passes or activation data.
Kurtosis as a Signal: The paper establishes that high kurtosis in vocabulary-projected weights is a robust statistical signal for monosemanticity, validated across different models (Llama-3.1, Gemma-2, OLMo) and training stages.
Vocabulary Channels: The method recovers "vocabulary channels"—sparse directions in weight space that correspond to specific concepts (e.g., "temporal connectors," "negative numbers," "GUI programming").
Superior Performance: ROTATE outperforms state-of-the-art SAE-based baselines (Gemma Scope, Llama Scope) in both faithfulness (predicting neuron behavior) and completeness (covering the neuron's activation spectrum).

4. Experimental Results

The authors evaluated ROTATE on Gemma-2-2B-it and Llama-3.1-8B-Instruct.

A. Faithfulness (Input and Output)

Input-Side: ROTATE channels were tested by generating synthetic inputs that match the channel description. These inputs consistently activated the target neuron significantly more than neutral inputs. ROTATE achieved faithfulness scores of 0.46–0.71, outperforming SAE baselines (0.33–0.49).
Output-Side (Causal Ablation): The authors ablated specific channels from the output weight vector ( $w_{out}$ $w_{o u t}$ ) and measured the impact on the model's generation.
- Self-Ablation: Removing a channel suppressed the specific tokens associated with that channel by ~87–90%.
- Cross-Ablation: Removing a different channel had negligible effect or slightly increased the frequency of the target tokens.
- This confirms that the channels are causally responsible for specific concept promotion and are well-separated.

B. Completeness

The authors tested whether the set of discovered channels could explain the neuron's behavior on diverse inputs.
An LLM judge was asked to match an activating input to the correct channel description among distractors.
ROTATE achieved 49–60% accuracy, significantly outperforming SAE baselines (36–49%) and far exceeding random chance (20%). This suggests ROTATE captures a broader spectrum of the neuron's functionality.

C. Neuron Descriptions

By aggregating the descriptions of a neuron's multiple channels, the authors generated comprehensive natural language descriptions of the neuron's function.
In head-to-head comparisons against activation-based baselines (MaxAct++ and MaxAct+VocabProj), ROTATE descriptions won 60–69% of the time, particularly for inputs that were not the absolute top-activating examples (ranks 100–500). This demonstrates that weight-based decomposition captures concepts that activation-based methods miss.

5. Significance and Conclusion

Scalability: Because ROTATE requires no forward passes, it is computationally efficient and scalable to massive models and datasets where collecting activation data is prohibitive.
Fine-Grained Interpretability: It provides a "building block" for understanding LLMs by breaking down polysemantic neurons into their constituent monosemantic parts directly from the weights.
New Paradigm: It shifts the focus from analyzing what the model does (activations) to what the model knows (weights), offering a more fundamental view of the model's internal knowledge representation.

The authors conclude that vocabulary channels are a scalable, fine-grained unit of analysis for mechanistic interpretability, offering a promising path toward more accurate circuit discovery and concept manipulation in language models. The code is released at https://github.com/AsafAvr/rotating-neurons.