💬 NLP

Constructing Interpretable Features from Compositional Neuron Groups

This paper proposes using semi-nonnegative matrix factorization (SNMF) to decompose MLP activations into sparse, interpretable features composed of co-activated neurons, demonstrating that this approach outperforms sparse autoencayers and supervised baselines in causal steering while revealing a hierarchical structure of concept representations in large language models.

Original authors: Or Shafran, Atticus Geiger, Mor Geva

Published 2026-05-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Or Shafran, Atticus Geiger, Mor Geva

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Large Language Model (LLM) like a massive, complex orchestra. For a long time, scientists trying to understand how this orchestra works have focused on individual musicians (neurons). They thought, "If I can figure out what one violinist is doing, I'll understand the music."

However, the paper argues that this approach is flawed. In reality, musicians often play together in groups to create specific sounds. A single violinist might play a sad note in one song and a happy note in another. The "meaning" isn't in the violinist alone; it's in the combination of musicians playing at the same time.

Here is the paper's story, broken down into simple concepts:

1. The Problem: The "Dictionary" Was Too Messy

Previously, researchers tried to find these musical groups using a tool called a Sparse Autoencoder (SAE). You can think of an SAE as a very ambitious librarian trying to organize a chaotic library. It tries to invent new "shelves" (features) from scratch to sort the books (data).

The problem? The librarian sometimes creates shelves that don't make sense to humans, or the books on the shelves don't actually belong there. When researchers tried to use these shelves to control the orchestra (make it play a specific song), the librarian's system often failed. It was like trying to steer a ship using a map that was drawn from scratch without looking at the ocean.

2. The Solution: The "Compositional" Approach

The authors propose a new method called SNMF (Semi-Nonnegative Matrix Factorization).

Instead of inventing new shelves, SNMF looks at the orchestra's sheet music (the model's internal math) and asks: "Which musicians are playing together right now?"

The Analogy: Imagine the orchestra is playing a song about "Rain." SNMF doesn't invent a new "Rain" button. Instead, it notices that the Flute, the Cello, and the Timpani are all playing loudly at the same time. It groups them together and says, "This specific trio is the 'Rain' feature."
The Key Difference: Because SNMF builds its groups directly from the musicians who are actually playing, it knows exactly which notes (inputs) triggered that group. It's like having a direct line to the conductor saying, "When you see these three instruments, you know it's raining."

3. The Results: Better Control and Clearer Meaning

The researchers tested this new method on several famous AI models (like Llama 3.1 and Gemma 2). They compared it to the old "librarian" method (SAE) and a very strict, supervised method (where humans tell the computer exactly what to look for).

The "Steering" Test: They tried to "steer" the AI to talk about specific topics (like "police" or "history"). SNMF was much better at this. It could turn the AI's output toward the desired topic more effectively than the old methods, while still keeping the sentences grammatically correct and fluent.
The "Detection" Test: They checked if the AI could recognize when a topic was being discussed. SNMF performed just as well as the best existing methods at spotting these concepts.

4. The Big Discovery: The "Lego" Structure

The most fascinating part of the paper is what they found about how these groups are built.

They discovered that the AI builds complex ideas by stacking simpler ones, like Lego bricks.

The Analogy: Imagine you have a group of neurons that represents "Day." Then, you have a slightly different group that represents "Weekday." The "Weekday" group is just the "Day" group plus a few extra neurons that say, "No, not Saturday or Sunday."
The Hierarchy: By applying their method recursively (repeatedly), they saw a tree structure.
- Bottom Level: Specific neurons for "Monday," "Tuesday," etc.
- Middle Level: A group for "Weekdays" (combining Mon-Fri).
- Top Level: A group for "Days of the Week" (combining weekdays and weekends).

This shows that the AI doesn't just have random neurons; it has a hierarchical architecture. It reuses the same core "building blocks" (neurons) to create more complex ideas. For example, the core neurons for "Day" are reused for "Monday," "Tuesday," and "Weekend," with only a few extra neurons added to distinguish them.

Summary

The paper claims that by looking at groups of neurons working together (rather than individuals) and using a math tool that respects how the model actually calculates things, we can:

Find concepts that humans can easily understand.
Control the AI's output much more effectively than before.
See that the AI builds complex ideas by combining simpler, reusable parts, much like building a house with bricks.

The authors conclude that this method is a simple, effective way to "dissect" how AI models think, revealing that they are organized in a logical, hierarchical way that we can now see and use.

Technical Summary: Constructing Interpretable Features from Compositional Neuron Groups

Problem Statement
A central challenge in mechanistic interpretability is identifying the appropriate units of analysis within Large Language Models (LLMs) that causally explain model outputs. While early research focused on individual neurons, evidence suggests neurons often encode multiple concepts (polysemanticity), motivating a shift toward analyzing directions in activation space. Current unsupervised approaches rely on Sparse Autoencoders (SAEs) trained on residual stream activations to learn feature directions from scratch. However, SAEs face significant limitations: they often struggle in causal evaluations (steering), lack intrinsic interpretability because their learned features are not explicitly tied to the model's internal computations, and their features are not constrained to the model's representation space. There is a need for a method that discovers interpretable features in an unsupervised manner while maintaining a direct causal link to the model's mechanisms.

Methodology
The authors propose decomposing Multi-Layer Perceptron (MLP) activations directly using Semi-Nonnegative Matrix Factorization (SNMF). Unlike SAEs, which learn new directions, this method represents features as sparse linear combinations of existing co-activated neurons within the MLP.

Formulation: Given a matrix $A \in \mathbb{R}^{d_a \times n}$ of MLP activations (where $d_a$ is the inner dimension and $n$ is the number of tokens), the method factorizes $A$ into two matrices:
- $Z \in \mathbb{R}^{d_a \times k}$ : A matrix of $k$ MLP features, where each column defines a sparse linear combination of neurons.
- $Y \in \mathbb{R}^{k \times n}_{\geq 0}$ : A non-negative coefficient matrix mapping inputs to features.
  The decomposition follows $A \approx ZY$ .
Interpretability Mechanism:
- Sparsity: The columns of $Z$ are constrained to be sparse using a hard winner-take-all operator, ensuring each feature activates only a small subset of neurons.
- Causal Grounding: The non-negative matrix $Y$ explicitly encodes which tokens contributed to the creation of a feature, allowing for direct assignment of textual concept labels based on high-activating contexts.
- Mapping to Residual Stream: Each feature vector $z_i$ in the activation space is mapped to the residual stream via the MLP output weights ( $W_V$ ), defining a residual stream feature $f_i = W_V z_i$ . This ensures features are grounded in the model's actual computational pathways.
Optimization: The method uses multiplicative updates (Ding et al., 2010) to minimize the Frobenius norm between $A$ and $ZY$, with sparsity enforced on $Z$ .

Key Contributions

SNMF for MLP Decomposition: The paper introduces an unsupervised method to decompose MLP activations into interpretable features defined by groups of co-activated neurons, rather than learning arbitrary directions from scratch.
Causal Steering Superiority: The authors demonstrate that SNMF-derived features outperform state-of-the-art SAEs (GemmaScope, Llamascope) and a strong supervised baseline (Difference-in-Means) in causal steering tasks, while performing comparably in concept detection.
Revealing Hierarchical Composition: Through recursive SNMF, the authors show that MLP features are compositional. Fine-grained concepts (e.g., specific weekdays) are formed by combining shared "core" neurons with exclusive neurons. This reveals a hierarchical structure where broader concepts (e.g., "weekday") are composed of more specific sub-concepts, effectively inverting the "feature splitting" phenomenon often observed in SAEs.
Neuron Reuse: The analysis confirms that semantically related features share a core set of neurons, with exclusive neurons controlling the promotion or suppression of specific sub-concepts.

Results
Experiments were conducted on Llama 3.1-8B, Gemma 2-2B, and GPT-2 Small.

Concept Detection: SNMF features achieved positive detection scores for the majority of features (>75%), indicating they activate for concept-related inputs and remain inactive for neutral inputs. Performance was comparable to or exceeded that of SAEs trained on MLP outputs.
Concept Steering: In causal steering evaluations (measuring the ability to steer model generation toward a concept while preserving fluency), SNMF consistently outperformed SAEs across all layers. It also frequently matched or exceeded the performance of the supervised Difference-in-Means baseline.
Hierarchical Analysis: Recursive application of SNMF successfully merged fine-grained features (e.g., "Monday," "Tuesday") into higher-level concepts (e.g., "Weekday," "Weekend"). Causal interventions confirmed that amplifying "core" neurons promoted all related tokens, while amplifying "exclusive" neurons promoted specific tokens and suppressed others.

Significance and Claims
The paper claims that SNMF provides a simple and effective tool for identifying interpretable features in LLMs. By directly decomposing MLP activations, the method ties features to the specific model components responsible for their formation. The results suggest that the MLP operates via additive updates composed of interpretable neuron sets. This work positions SNMF as a viable alternative to SAEs, offering better causal performance and intrinsic interpretability without requiring supervision. Furthermore, the discovery of hierarchical neuron composition provides a new lens for understanding how LLMs construct complex concepts from reusable neural building blocks, extending the functional role of neurons beyond single-concept encoding.

The authors acknowledge limitations, including the current focus on feature counts $k < 500$ and the reliance on LLMs as judges for evaluation, though they report strong agreement with human judgments. They suggest future work could explore scaling to larger $k$ values and utilizing projected gradient descent for optimization.