🤖 machine learning

Do Sparse Autoencoders Capture Concept Manifolds?

This paper establishes a theoretical framework demonstrating that Sparse Autoencoders can capture concept manifolds either globally or locally, but often fail to do so effectively due to a "dilution" regime that fragments these structures, thereby motivating a shift in interpretability research from isolated directions to coherent groups of features.

Original authors: Usha Bhalla, Thomas Fel, Can Rager, Sheridan Feucht, Tal Haklay, Daniel Wurgaft, Siddharth Boppana, Matthew Kowal, Vasudev Shyam, Jack Merullo, Atticus Geiger, Ekdeep Singh Lubana

Published 2026-05-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Usha Bhalla, Thomas Fel, Can Rager, Sheridan Feucht, Tal Haklay, Daniel Wurgaft, Siddharth Boppana, Matthew Kowal, Vasudev Shyam, Jack Merullo, Atticus Geiger, Ekdeep Singh Lubana

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand how a giant, complex machine (like a Large Language Model) thinks. For a long time, scientists believed the machine's "thoughts" were like a set of distinct, independent switches. If you wanted to talk about "age," you just flipped the "age switch." If you wanted to talk about "temperature," you flipped the "temperature switch." This idea is called the Linear Representation Hypothesis.

However, this new paper argues that the machine's thoughts aren't actually made of isolated switches. Instead, they are more like smooth, curved roads or manifolds.

Here is a breakdown of what the paper found, using simple analogies:

1. The Problem: Roads vs. Switches

Think of a concept like "Temperature." In the old view, there was one specific direction in the computer's brain for "Hot" and another for "Cold." But the paper shows that in reality, temperature is a continuous curve. You can go from freezing to boiling smoothly. The machine's internal representation of this isn't a single line; it's a curved path where "Warm" is right next to "Hot," and "Cold" is right next to "Cool."

The paper calls these curved paths Manifolds.

2. The Tool: Sparse Autoencoders (SAEs)

To study these thoughts, researchers use a tool called a Sparse Autoencoder (SAE). You can think of an SAE as a translator that tries to break the machine's complex thoughts down into a list of simple, understandable "features" or "atoms."

The big question the paper asks is: Can this translator successfully map out those smooth, curved roads (manifolds), or does it just see a bunch of disconnected switches?

3. The Discovery: The Translator is "Diluted"

The paper finds that current SAEs do not capture these curved roads perfectly. Instead of finding one smooth road, the SAE breaks the road into tiny, fragmented pieces.

The authors describe three ways an SAE could handle a curved road, but only one works well, and current models mostly do the third, messy one:

The Ideal Way (Global Capture): Imagine the SAE finds a small, perfect team of 5 "atoms" that, when combined, can draw the entire curved road. This is efficient and clean.
The "Shattering" Way (Local Tiling): Imagine the SAE assigns one specific atom to "Freezing," another to "Chilly," another to "Cool," and so on. Each atom is a tiny tile covering a small patch of the road. This works, but you need hundreds of atoms to cover the whole road.
The "Dilution" Way (The Reality): This is what the paper found in real models. The SAE is confused. It uses too many atoms, and they overlap in a messy way. Some atoms cover "Freezing," others cover "Chilly," but they also overlap with "Cool" and "Cold" in a redundant, confusing jumble.

The authors call this state "Dilution." The geometry is there, but it's spread out so thinly across so many features that if you look at just one feature, it doesn't make sense. It's like trying to understand a painting by looking at a single, blurry pixel rather than the whole picture.

4. The Solution: Looking for Groups, Not Individuals

Because the SAEs are "diluted," you can't just look at one feature and say, "Ah, this is the 'Temperature' feature."

Instead, the paper suggests we need to look for groups of features that work together.

The Analogy: Imagine a crowd of people trying to form a human chain to represent a curved line. If you look at one person, they are just standing there. But if you look at the group, you see the curve.
The Method: The authors developed a new way to find these groups. They use a method inspired by physics (called an Ising Model) to look at which features "fire" (activate) together or cancel each other out.
- If two features are part of the same "road," they might fire together (positive connection).
- If they represent different parts of the road that don't overlap, they might never fire at the same time (negative connection).

By mapping these connections, they can reconstruct the smooth, curved roads that the SAE had broken apart.

5. Why This Matters

The paper concludes that we need to change how we interpret AI.

Old View: Meaning is found in single, isolated directions (like a single switch).
New View: Meaning is found in geometric shapes (like a curved road) formed by groups of features working together.

The authors warn that if we keep trying to interpret AI by looking at single features, we might miss the point or even invent fake meanings (hallucinations). To truly understand the machine, we have to stop looking for individual switches and start looking for the curved roads they collectively build.

In short: The machine thinks in smooth curves, but our current tools break those curves into messy, overlapping fragments. To understand the machine, we need to learn how to put those fragments back together into the original shape.

1. Problem Statement

Sparse Autoencoders (SAEs) are the standard tool for extracting interpretable features from Large Language Models (LLMs). Their operation relies on the Linear Representation Hypothesis (LRH), which assumes that abstract concepts correspond to independent, linear directions (rays) in activation space.

However, recent evidence suggests that many concepts (e.g., time, temperature, color, political bias) are not linear but are organized along low-dimensional, curved manifolds. The paper addresses a critical gap:

The Mismatch: SAEs are designed to find linear directions, but the underlying geometry of concepts is often nonlinear and continuous.
The Core Question: Do SAEs capture these manifolds? If so, how? And if they do, why is the manifold structure often invisible when inspecting individual features?

2. Methodology

The authors develop a theoretical framework and validate it through synthetic benchmarks and empirical analysis on Llama 3.1-8B.

A. Theoretical Framework: Additive Mixture of Manifolds

The authors formalize a model where representations are an additive superposition of manifolds ( $x = \sum f_i(m_i)$ ). They define three distinct regimes for how an SAE might represent a manifold:

Subspace Capture (Ideal): A small, fixed group of atoms spans the entire manifold. The same atoms fire for every point on the manifold, and their coefficients vary continuously.
Tiling (Shattering): The manifold is partitioned into disjoint regions. Different atoms fire for different regions (mutually exclusive), acting like localized detectors.
Tiling (Dilution): The manifold is covered by a large, redundant set of atoms. Multiple atoms fire for the same region, but no single compact group spans the whole manifold. This results in "mixed selectivity."

B. Synthetic Validation

The authors constructed a controlled benchmark with 48 synthetic manifolds (circles, spheres, tori, Möbius strips, helices, etc.) embedded in high-dimensional space. They trained SAEs with varying sparsity budgets ( $k$ ) to observe which regime emerges.

Metric: They introduced a Restricted $R^2$ score to measure how well a small subset of atoms reconstructs a specific manifold.
Discovery: They found that SAEs rarely achieve "Subspace Capture." Instead, they often fall into a Dilution regime, where many redundant atoms tile the manifold.

C. Unsupervised Discovery via Ising Models

To recover the hidden manifold structure from the fragmented SAE features, the authors propose using a pairwise Ising model on binarized feature activations.

Logic: Instead of looking at geometric similarity (decoder directions), they analyze co-activation statistics.
Mechanism: Atoms that tile the same manifold will exhibit strong statistical dependencies (either positive coupling if they co-fire, or negative coupling if they are mutually exclusive/shattered).
Algorithm: Fit an Ising model to the binary codes to extract a coupling matrix $J$ . Community detection on $|J|$ reveals groups of atoms that collectively represent a manifold.

D. Empirical Analysis on LLMs

The framework was applied to Llama 3.1-8B (Layer 19) using five different SAE architectures (Standard, JumpReLU, TopK, BatchTopK, Matryoshka).

Manifolds Tested: Continuous concepts like Age, Temperature, Days of the Week, Colors, and Political Bias.
Steering Experiments: The authors verified that steering between centroids of these manifolds produces smooth, predictable changes in token probabilities, confirming the manifolds are causally relevant.

3. Key Results

A. SAEs Operate in the "Dilution" Regime

Contrary to the ideal of "Subspace Capture," empirical results show that current SAEs do not allocate a compact group of atoms to span a manifold.

Fragmentation: Manifolds are represented by a large, partially redundant set of features.
Piecewise-Linear Approximation: Individual SAE features act as localized detectors (tuning curves) covering small patches of the manifold. The union of these patches approximates the smooth geometry, but no single feature captures the global structure.
Sparsity Sweet Spot: There is a specific sparsity level where "Capture" is possible, but standard SAE training often defaults to "Dilution" or "Shattering" due to architectural constraints and optimization dynamics.

B. Feature Tuning Curves

Analysis of feature activations reveals "tuning curves" reminiscent of biological population coding (e.g., place cells).

For the "Year" manifold, features activate periodically (e.g., every 10 years for the "ones" digit) or cover specific decades.
No single feature encodes the full concept; the concept is encoded by the population code of the group.

C. Unsupervised Discovery Success

The Ising-based pipeline successfully recovered known manifolds (Temperature, Colors, Days) and discovered novel manifolds (e.g., a manifold encoding epistemic uncertainty in scientific texts) without supervision.

Superiority over Cosine Similarity: Clustering based on decoder cosine similarity failed to recover manifolds because atoms spanning a manifold can be orthogonal. Ising couplings (conditional independence) successfully identified the functional groups.

4. Key Contributions

Formalization of Manifold Capture: Defined mathematically what it means for an SAE to capture a manifold, distinguishing between Subspace Capture (global) and Tiling (local/shattered/diluted).
Identification of the "Dilution" Regime: Demonstrated that current SAEs typically represent manifolds through fragmented, redundant collections of atoms rather than coherent subspaces. This explains why individual features often appear "mixed" or lack clear semantic meaning.
Ising-Based Discovery Method: Proposed and validated a post-hoc unsupervised method using Ising models to group features into coherent manifolds, overcoming the limitations of direction-based interpretability.
Reframing Interpretability: Argued that the unit of interpretability should shift from individual directions to geometric objects (manifolds).

5. Significance and Implications

Explaining SAE Instability: The paper explains why learned dictionaries are unstable across runs. If concepts are manifolds, there are multiple valid "tilings" (bases) to represent the same geometry. Different initializations lead to different but equally valid decompositions.
Steering Limitations: It explains why steering via individual features is often brittle. Intervening on a single feature pushes the representation off the manifold rather than moving smoothly along it.
Future Directions:
- Featurization: Future SAE architectures should treat geometric objects as primitives, not just directions.
- Analysis: Interpretability tools must analyze groups of features (subspaces) rather than isolated atoms.
- Workaround: Until new architectures are developed, the Ising-based post-hoc analysis is the most reliable method for recovering meaningful concept structures from existing SAEs.

In summary, the paper argues that while SAEs successfully preserve the geometry of neural representations, they do so in a fragmented, diluted manner. To truly understand these models, researchers must move beyond the "Linear Representation Hypothesis" and adopt a geometric perspective that treats concepts as low-dimensional manifolds tiled by populations of features.