💬 NLP

Why Linear Interpretability Works: Invariant Subspaces as a Result of Architectural Constraints

The paper proposes that linear interpretability methods succeed because transformer architectures force semantic information into context-invariant linear subspaces through their linear communication interfaces, a phenomenon they formalize as the "Invariant Subspace Necessity" theorem.

Original authors: Andres Saurez, Yousung Lee, Dongsoo Har

Published 2026-02-11

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Andres Saurez, Yousung Lee, Dongsoo Har

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand how a massive, complex city works. You see millions of people moving, cars driving, and lights flashing. It looks like total chaos—a "nonlinear" mess. But then, you notice something: every time someone wants to go to the airport, they follow a specific highway. Every time someone wants to buy food, they head toward a grocery store.

Even though the city is chaotic, the infrastructure (the roads and signs) forces people to move in predictable, straight lines to get where they need to go.

This paper explains that Large Language Models (like ChatGPT) work exactly like this city.

The Big Mystery: Why is "Chaos" so Organized?

AI models are incredibly complex. They are made of billions of mathematical connections that are "nonlinear"—meaning they are twisty, curvy, and incredibly complicated.

For a long time, researchers have noticed something weird: even though the "brain" of the AI is a twisty mess, if you look at it with a simple "straight ruler" (called a Linear Probe), you can easily find things. You can point a straight line at a group of neurons and say, "Aha! This direction represents 'France'."

The big question was: Why? Why does a twisty, complicated brain allow us to use a simple, straight ruler to find meaning?

The Discovery: The "Highway" Constraint

The authors of this paper argue that it’s not a coincidence. It’s because of the architecture of the AI.

Think of the AI as a giant communication network. For the AI to "talk" to itself and pass information from one layer to the next, it has to use certain "interfaces" (like the Attention mechanism). These interfaces act like straight highways.

The authors prove a mathematical rule: If you want to send a specific piece of information (like the concept of "France") through a straight highway, that information must travel in a straight line.

If the information traveled in a wiggly, unpredictable way, it would get lost or garbled when it hit the "highway" interface. Therefore, to be efficient, the AI is forced to organize its "concepts" into straight, predictable directions.

The "Self-Reference" Trick: The Name is the Map

This leads to a mind-blowing discovery the authors call the Self-Reference Property.

Imagine you are in a massive, dark library. You are looking for the concept of "Apples." Instead of wandering aimlessly, you find a single book titled "Apple." The authors found that in an AI, the word "Apple" itself acts like a compass needle.

If you look at the mathematical "direction" of the word "Apple," that direction points exactly toward every other thing related to apples—like "fruit," "red," "crunchy," or "Newton." The word doesn't just represent the concept; it provides the map to find the concept.

Why This Matters (The "So What?")

This paper is a huge deal for three reasons:

It explains why our tools work: We now know that "Linear Probes" (simple rulers) work not because the AI is simple, but because the AI's "roads" force it to be organized.
Zero-Shot Understanding: It means we can find new concepts in an AI without "teaching" it. We can just grab a word, use it as a compass, and see what else the AI thinks is in that direction.
Unifying the Science: It connects two different ways of looking at AI (Sparse Autoencoders and Linear Probes), proving they are both looking at the same "highways."

In short: The AI's brain is a wild, swirling storm, but because it has to communicate through straight roads, it builds organized "highways of meaning" that we can finally map out.

Technical Summary: Why Linear Interpretability Works

Title: Why Linear Interpretability Works: Invariant Subspaces as a Result of Architectural Constraints
Authors: Andres Suarez, Yousung Lee, Dongsoo Har (KAIST)

1. The Problem: The "Linearity Puzzle" in Nonlinear Systems

Modern Transformer models are deep, highly nonlinear systems. However, a consistent empirical regularity in mechanistic interpretability is that simple linear methods—such as linear probes, Sparse Autoencoders (SAEs), and activation steering—are remarkably successful at extracting and manipulating semantic features.

The fundamental theoretical puzzle is: Why does a nonlinear architecture yield such reliable, low-dimensional linear structures? While previous research has attributed this to training objectives (next-token prediction) or optimization biases (gradient descent), this paper argues that the architecture itself necessitates this linear structure.

2. Methodology and Theoretical Framework

The authors propose a geometric framework centered on the concept of Identity-Projection.

A. Core Assumptions:

Additive Residual Stream: Information is updated additively across layers.
Linear Communication Interfaces: The Attention OV (Output-Value) circuits and the Unembedding matrix ( $W_U$ ) act as linear maps on the residual stream.
Shared Parameters: Parameters are applied across all token positions.

B. Key Theoretical Results:

Invariant Subspace Necessity Theorem: The authors prove that if a semantic feature is "communicable" (meaning it can be recovered through a linear interface), its representation must reside in a context-invariant subspace. This means that while the magnitude of a feature (e.g., "how much 'France' is in this sentence") changes with context, the geometric direction remains stable.
Capacity Constraints & Factorization: Because the vocabulary size $|V|$ is much larger than the hidden dimension $d$ , tokens cannot have unique orthogonal directions. To minimize interference, the model must use factorized representations, where tokens are sparse combinations of shared, low-dimensional feature directions.
The Self-Reference Property: A pivotal derivation stating that a token (e.g., "France") does not just represent a concept; its own hidden state $h_t$ serves as the geometric reference vector for that feature's direction in the activation space.

C. Empirical Validation Methods:
The authors test these theories using three types of probes:

Zero-Shot Probes: Using raw token directions to classify implicit instances without training.
Unsupervised Probes: Training a linear probe using only class tokens to see if it generalizes to instances.
SAE Probes: Using latents from Sparse Autoencoders (trained only on instances) to see if they align with token directions.

3. Key Contributions

Architectural Explanation: Shifts the explanation of linear interpretability from "how models are trained" to "how models are built," proving that linear interfaces force features into invariant subspaces.
Self-Reference Property: Provides a mathematical basis for why we can use specific tokens as "anchors" to find semantic directions without labeled data.
Unification of Methods: Demonstrates that linear probes and SAEs are not competing methods but are different ways of accessing the same underlying invariant geometric structure.
Geometric View of Polysemy: Explains polysemy (one token, multiple meanings) and partial synonymy (multiple tokens, shared meanings) as two sides of the same directional superposition.

4. Results

The authors conducted extensive experiments across eight semantic domains (Taxonomic, Affective, Stylistic, etc.) and four model families (LLaMA3, Mistral, GPT-2, LLaMA3.2).

High Accuracy: All methods (Zero-shot, Unsupervised, and SAE) achieved strong classification accuracy. For example, in the "Animals" task, unsupervised probes and SAEs performed comparably to supervised baselines.
Validation of Self-Reference: In LLaMA3-8B, between 89% and 98% of attention heads showed that a class token's direction aligned more strongly with its own class instances than with other classes.
SAE Alignment: SAEs trained purely on context (without seeing class tokens) successfully recovered the same directions used by token-based probes, as evidenced by high overlap in top- $k$ latent dimensions.
Generalization: The unsupervised probe trained on only half of a dataset (e.g., odd months) successfully classified the remaining half (even months), proving the learned structure is truly semantic and not just memorized.

5. Significance

This paper provides a principled, mathematical foundation for the field of mechanistic interpretability. By proving that transformers are "interpretable by design," it moves the field away from empirical observation toward a formal science.

Practical Implications:

Scalable Interpretability: The Self-Reference Property suggests we can discover "circuits" and semantic directions in massive models using unsupervised, token-based methods rather than expensive human labeling.
Model Steering: It provides a theoretical guarantee for why "activation steering" (adding a direction to a hidden state) works: because those directions are stable, invariant components of the model's internal language.