💬 NLP

Decomposing Representation Space into Interpretable Subspaces with Unsupervised Learning

This paper introduces an unsupervised method called neighbor distance minimization (NDM) that successfully decomposes neural model representation spaces into interpretable, non-basis-aligned subspaces corresponding to distinct abstract concepts and circuit variables, as demonstrated through qualitative analysis and quantitative experiments on GPT-2 and 2B-scale models.

Original authors: Xinting Huang, Michael Hahn

Published 2026-05-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xinting Huang, Michael Hahn

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a neural network (like the brain of an AI) as a massive, high-dimensional room filled with invisible "thoughts." When the AI processes a sentence, it doesn't just store one idea; it stores thousands of different details at once: the current word, the position of that word, the topic of the conversation, and the grammar rules.

The problem is that all these thoughts are jumbled together in a giant, messy pile. If you wanted to understand how the AI works, it's like trying to find a specific thread in a ball of yarn where every color is mixed up.

The Big Idea: Untangling the Yarn
This paper introduces a new method called Neighbor Distance Minimization (NDM). Think of the AI's "thought room" as a crowded dance floor where everyone is dancing in a chaotic mix. The researchers wanted to find a way to separate the dancers into distinct groups based on what they are actually doing, without being told what to look for.

They discovered that if you organize the room so that dancers who are doing similar things stay close to each other, and dancers doing different things stay far apart, you naturally create separate "zones" or subspaces.

How It Works (The Analogy)
Imagine you have a box of mixed-up LEGO bricks: red ones, blue ones, and green ones.

The Old Way: Researchers usually had to guess which bricks went together or ask a human to sort them first (supervised learning).
The New Way (NDM): The researchers just told the box of bricks: "Keep bricks that are similar close to each other, and push the ones that are different far away."

Surprisingly, the box sorted itself! The red bricks naturally clumped together, the blue bricks formed their own group, and the green bricks did the same. The researchers didn't need to know what "red" or "blue" meant; the math of "staying close to your neighbors" did the sorting for them.

What They Found
Once they separated the room into these clean zones, they looked inside and found something amazing:

Specialized Zones: Each zone seemed to handle a specific type of information. One zone was dedicated entirely to "current words," another to "where we are in the sentence," and another to "the topic of the story."
Variables: These zones act like variables in a computer program. Just as a variable named x holds a number, these zones hold specific concepts. If the AI is talking about "Alice," the "name zone" holds "Alice." If it switches to "Bob," that same zone updates to "Bob," while the "topic zone" stays the same.
It Works on Big Models: They tested this on small toy models first, then on real AI models (like GPT-2 and 2-billion-parameter models). It worked everywhere. In the larger models, they even found separate zones that handled "what the model knows from its training" versus "what the model is reading right now in the prompt."

Why This Matters
Before this, trying to understand an AI was like trying to read a book where all the letters were scrambled. This method gives us a way to unscramble the letters into words and sentences.

The researchers call these new zones "subspace circuits." Instead of looking at individual neurons (which are like single pixels in a blurry image), we can now look at these whole zones (like clear, distinct words). This allows us to trace how the AI moves information from one part of its brain to another, giving us a much clearer map of how these intelligent machines actually think.

In Short:
The paper shows that you can teach an AI to organize its own messy thoughts into neat, separate folders just by asking it to keep similar thoughts close together. This reveals that the AI naturally builds a structured, understandable system of "variables" to process information, even without humans telling it how to do it.

Technical Summary: Decomposing Representation Space into Interpretable Subspaces with Unsupervised Learning

Problem Statement

Mechanistic interpretability aims to uncover the internal algorithms and circuits of neural networks. While significant progress has been made in identifying components (attention heads, MLPs), sparse features (via Sparse Auto-Encoders), and specific subspaces, existing methods face limitations. Component-based analysis often requires specialized counterfactual data for causal intervention. Sparse features provide interpretability but result in input-dependent circuits that modify model computation. Current subspace-based approaches typically rely on supervised learning guided by human-specified abstract causal models, requiring researchers to anticipate specific mechanisms beforehand.

The core question addressed by this paper is whether it is possible to find "natural," interpretable subspaces within the representation space of neural models in a purely unsupervised manner, without access to ground-truth features or human-defined hypotheses.

Methodology: Neighbor Distance Minimization (NDM)

The authors propose Neighbor Distance Minimization (NDM), an unsupervised method to partition the representation space into orthogonal, non-basis-aligned subspaces.

Core Intuition

The method is grounded in the hypothesis that neural representations exhibit mutual exclusiveness within feature groups. If a set of features is mutually exclusive (only one or none is active at a time), they can be superposed within a dedicated subspace. Conversely, features from different groups (which may co-occur) should be represented orthogonally to avoid interference.

Geometric Interpretation: In a correctly partitioned space, data points projected onto a specific subspace will lie on a few concentrated lines (representing the active features of that group). Consequently, each point will have a very close neighbor within that subspace.
Optimization Objective: NDM seeks an orthogonal matrix $R$ and a dimension configuration $c$ that minimizes the distance between each activation and its nearest neighbor within each resulting subspace.

Algorithm

Transformation: An orthogonal matrix $R$ rotates and reflects the model activations $h$ . The transformed space is partitioned into subspaces based on a dimension configuration $c = [d_1, \dots, d_S]$ .
Loss Function: The objective minimizes the average distance to the nearest neighbor within each subspace $s$ :
$\min_R \frac{1}{N} \sum_{s=1}^S \sum_{n=1}^N \text{dist}(\hat{h}_n^{(s)}, \hat{h}_{n^*}^{(s)})$
where $\hat{h}_{n^*}^{(s)}$ is the nearest neighbor of $\hat{h}_n^{(s)}$ in subspace $s$ .
Dimension Configuration: Since the number of subspaces and their dimensions are unknown, the method starts with small, equal-sized subspaces. During training, it periodically estimates the Mutual Information (MI) between subspaces using a KSG estimator. If the MI between two subspaces exceeds a threshold, they are merged. This process continues until no pairs exceed the threshold, effectively finding a partition where subspaces are as independent as possible.

Key Contributions and Results

1. Toy Model Validation

In a controlled toy setting where ground-truth feature groups are known, NDM successfully recovers the orthogonal subspaces dedicated to each feature group. The method minimizes the distance to nearest neighbors, leading to a partition where each subspace encodes only one feature group, validating the theoretical link between mutual exclusiveness, superposition, and subspace orthogonality.

2. Quantitative Evaluation on GPT-2 Small

The authors evaluated NDM on GPT-2 Small using subspace activation patching on known circuits (Indirect Object Identification and Greater-than circuits).

Metric: They measured the concentration of patching effects using the Gini coefficient. A high Gini coefficient indicates that a specific piece of information (e.g., the previous token) is concentrated in a single subspace.
Performance: NDM significantly outperformed baselines (Identity, Random, PCA, and SAE-feature clustering). NDM achieved an average Gini coefficient of 0.71, well above the 0.6 threshold for satisfactory concentration, whereas baselines ranged from 0.21 to 0.38.
Evidence: The results demonstrate that NDM finds subspaces that align with high-level "variables" in known circuits, such as the previous token or subject position, independent of the specific values those variables take.

3. Qualitative Analysis via InversionView

Using InversionView, the authors analyzed the preimages (inputs that map to similar activations) of the discovered subspaces.

Interpretability: Subspaces were found to encode distinct, interpretable aspects of the context, such as the current token, the previous token, token position, and the overall topic.
Consistency: The function of a subspace was consistent across different inputs (e.g., a specific subspace consistently encoded the current token), suggesting these subspaces act as stable "variables" for the model.

4. Scalability to Larger Models

The method was applied to Qwen2.5-1.5B and Gemma-2-2B to investigate knowledge conflict scenarios (parametric vs. context knowledge).

Findings: NDM identified separate subspaces mediating parametric knowledge routing and context knowledge routing. Patching experiments showed that specific subspaces had strong effects under counterfactual inputs designed to break one mechanism while leaving the other intact.
Significance: This suggests that even in larger models without known ground-truth circuits, NDM can uncover meaningful, separable functional structures.

Significance and Claims

The paper claims that NDM offers a new perspective on understanding model internals by decomposing the representation space into interpretable, input-independent subspaces without supervision.

New Basic Units: The authors propose that these subspaces can serve as fundamental units for subspace circuits. Unlike feature circuits which are input-dependent, subspace circuits operate on "variables" (groups of features) and can be constructed by analyzing model weights (e.g., which subspaces an attention head reads from and writes to).
Unsupervised Discovery: The method demonstrates that "natural" partitions exist in real-world models, driven by the statistical structure of the data (mutual exclusiveness) rather than human hypothesis.
Limitations: The authors acknowledge that current partitions may not be fine-grained enough (e.g., missing tiny subspaces for variables with small value sets) and that some subspaces remain hard to interpret. They note that the method relies on the Linear Representation Hypothesis and that optimization can sometimes get stuck in local minima.

In conclusion, the paper argues that unsupervised decomposition via neighbor distance minimization is a viable path toward discovering the "variables" and circuits that govern neural model behavior, providing a scalable and interpretable framework for mechanistic analysis.