🤖 machine learning

Unstable Features, Reproducible Subspaces: Understanding Seed Dependence in Sparse Autoencoders

This paper demonstrates that while individual Sparse Autoencoder features often lack reproducibility across training seeds, they collectively form stable, reproducible low-dimensional subspaces where stable features drive functional utility and unstable features merely reflect basis ambiguity rather than noise.

Original authors: Gleb Gerasimov, Timofei Rusalev, Nikita Balagansky, Daniil Laptev, Vadim Kurochkin, Daniil Gavrilov

Published 2026-06-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Gleb Gerasimov, Timofei Rusalev, Nikita Balagansky, Daniil Laptev, Vadim Kurochkin, Daniil Gavrilov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand how a giant, complex machine (like a modern AI) thinks. To do this, researchers use a tool called a Sparse Autoencoder (SAE). Think of an SAE as a translator that tries to break the AI's internal thoughts down into a list of simple, human-readable "features" or "concepts" (like "this sentence is about a cat" or "this word is capitalized").

However, there's a problem: if you run this translator twice with slightly different starting conditions (like rolling dice to pick the starting point), you often get a different list of features each time. Some features appear in both lists, but many others seem to vanish or change completely. This makes it hard to trust the translation.

This paper investigates why this happens and what it means. Here is the breakdown in simple terms:

1. The "Two Types of Features" Discovery

The researchers found that the features the AI learns fall into two distinct camps, like two different types of students in a classroom:

The "Stable" Students (The Reliable Ones):
- What they do: These features show up in almost every version of the translator. They are the heavy lifters. They carry the most important information needed to understand the AI's thoughts and predict what comes next.
- What they talk about: They explain big ideas, grammar rules, and meaningful phrases (e.g., "this is a question," "this is a name," "this describes a feeling").
- Analogy: Imagine a news anchor. No matter which camera angle you use, the anchor is always there, delivering the main story.
The "Unstable" Students (The Chameleons):
- What they do: These features are flaky. If you run the translator again, they often disappear or get replaced by something else. They don't carry much important weight on their own.
- What they talk about: They focus on tiny, superficial details. They trigger on specific punctuation marks, weird capitalization, or short letter combinations (e.g., "words starting with 'T' followed by a capital letter").
- Analogy: Imagine a student who only raises their hand when the teacher wears a red hat. If the teacher wears a blue hat, the student stays silent. They aren't reacting to the lesson; they are reacting to a random detail.

2. The "Group Hug" Theory (Subspaces)

You might think the "Unstable" students are just random noise or mistakes. The paper argues they aren't just noise.

The Finding: Even though individual unstable features keep changing, they tend to cluster together in a specific, smaller "group" or subspace.
The Analogy: Imagine a dance floor. The "Stable" features are the main dancers doing the choreography. The "Unstable" features are a group of people who keep changing partners and moves every time the music restarts. However, they are always dancing in the same corner of the room.
What this means: The AI knows there is a specific "corner" of information to be learned (like a specific type of punctuation pattern), but it can't agree on which specific dancer should represent it. It's a disagreement on the basis, not the existence of the concept.

3. The "Synthetic Proof"

To prove this, the researchers built a fake, simplified AI model (a "toy model") where they knew exactly how it worked.

They created a situation where the "truth" was a low-dimensional group (a small corner of the dance floor).
Result: The AI successfully learned the group (the corner), but it kept swapping out the individual dancers (features) every time it restarted. This confirmed that the instability is a natural result of how the AI tries to organize shared information, not just a bug.

4. The Solution: The "Best of All Worlds" Pool

The paper offers a clever way to fix the instability without losing the quality of the translation.

The Method: Instead of training one translator and hoping for the best, they trained many translators with different seeds. Then, they took the "Stable" features from all of them and combined them into one Master Dictionary.
The Result: This new Master Dictionary was much more stable (it didn't change as much between runs) and was just as good at explaining the AI's thoughts as the original, unstable ones.
The Takeaway: You don't have to choose between a stable dictionary and a good one. By pooling the reliable features from many attempts, you get both.

Summary

Unstable features aren't just broken; they are real but fragile patterns that the AI struggles to pin down to a single label.
Stable features are the core, meaningful concepts.
The Instability comes from the AI having many ways to describe the same small group of patterns, leading to a "basis ambiguity" (a disagreement on the label, not the concept).
The Fix: Combine the best, most reliable features from many different training runs to build a stronger, more consistent translator.

Technical Summary: Unstable Features, Reproducible Subspaces

Problem Statement

Sparse Autoencoders (SAEs) are a primary tool for mechanistic interpretability, aiming to decompose neural network activations into sparse, human-interpretable features. However, a critical limitation is seed dependence: SAEs trained on identical data and hyperparameters but with different random seeds often learn substantially different feature sets. This raises a fundamental question: when a specific feature fails to reappear in a new training run, is the underlying direction absent from the new dictionary, or has the SAE simply learned a different basis for the same region of activation space? Distinguishing between these possibilities is vital because automatic interpretation methods can assign plausible explanations to features even in non-canonical or random settings, meaning an "interpretable-looking" feature may not be a reproducible unit of analysis.

Methodology

The authors investigate this problem through feature stability, defined as the probability that a feature reappears in an independently trained SAE under a cosine-similarity matching rule.

Stability Estimation: The authors train $N+1$ SAEs on identical data with different seeds. One run serves as an anchor. For each anchor feature, they count how many other runs contain a feature with cosine similarity $\ge \theta$ (default $\theta=0.7$ ). This yields an empirical reappearance probability $\hat{p}_i$ for each feature.
Classification: Features are categorized as stable ( $\hat{p} \ge 1-\epsilon$ ) or unstable ( $\hat{p} \le \epsilon$ ), with $\epsilon=0.05$ .
Comparative Analysis: Stable and unstable features are compared across multiple dimensions:
- Functional Impact: Measured via reconstruction quality (Explained Variance) and downstream next-token loss under activation patching.
- Activation Statistics: Frequency and magnitude of activation.
- Token Structure: Token entropy (diversity of tokens triggering the feature).
- Automatic Interpretation: Analysis of explanations generated by SAEBench.
Geometric Analysis: The authors analyze the decoder space geometry, computing the effective rank of subspaces spanned by stable vs. unstable features and testing the cross-seed reproducibility of these subspaces using Singular Value Decomposition (SVD) and logistic regression classifiers.
Synthetic Validation: A controlled low-rank synthetic model is constructed where ground-truth features exist in a shared low-rank subspace. SAEs are trained on this data to verify if seed-dependent basis ambiguity within a shared subspace can reproduce the observed instability patterns.
Feature Pooling: To test if stability can be improved without sacrificing reconstruction quality, the authors construct new SAEs by pooling unique features from multiple independent runs, selecting those with the highest reappearance probabilities.

Key Results

1. Functional Asymmetry

There is a pronounced functional asymmetry between stable and unstable features:

Stable Features: Carry the majority of the reconstruction- and prediction-relevant signal. They exhibit higher activation frequencies, heavier high-magnitude tails, and higher token entropy (covering broader lexical and conceptual patterns).
Unstable Features: Have weak marginal impact. Masking a large number of unstable features causes only a slight degradation in reconstruction quality and next-token loss, whereas masking far fewer stable features causes substantial degradation. They are dominated by low-frequency surface-form triggers (e.g., punctuation, specific substrings, capitalization patterns) and have lower token entropy.

2. Geometric Structure: Reproducible Subspaces

While individual unstable features are non-reproducible across seeds, they are not pure noise.

Low-Rank Concentration: Unstable features concentrate in lower-dimensional subspaces compared to stable features (effective rank reduction of ~20–27%).
Subspace Reproducibility: Although the specific basis vectors (individual features) change across seeds, the subspaces they span are highly reproducible. A classifier trained to distinguish unstable features in one seed transfers effectively to other seeds, and the top singular subspaces of one seed accurately approximate those of others.
Synthetic Confirmation: The synthetic low-rank model confirms this mechanism: low-rank ground-truth features are recovered at the subspace level but fail to align as individual latents across seeds due to basis ambiguity within the shared region.

3. Construction of Stable SAEs

By pooling unique features from multiple independent runs and selecting those with the highest reappearance probabilities, the authors construct "seed-robust" SAEs.

These constructed SAEs recover near-baseline Explained Variance (EV) after brief tuning.
Crucially, this approach demonstrates that stability and reconstruction quality need not trade off in this construction setting; high stability coincides with high EV.
These pooled SAEs remain competitive with standard TopK SAEs on SAEBench interpretability metrics.

4. Robustness and Baselines

Training Scale: Increasing training tokens reduces instability initially but approaches a non-zero plateau (~8% unstable features), indicating that instability is not merely a result of under-training.
Random Models: When SAEs are trained on activations from a randomly initialized transformer (a "dead-salmon" baseline), the stable fraction collapses, and instability dominates. This confirms that the stable features in trained models reflect genuine structure, not just artifacts of the SAE objective.
SAE Variants: Different sparsity mechanisms (TopK, BatchTopK, JumpReLU, Vanilla) show varying degrees of stability, but the bimodal distribution of stable/unstable features persists across most variants.

Significance and Claims

The paper argues that unstable features in SAEs are not merely failed or noisy latents. Instead, they reflect a reproducible low-dimensional structure within the activation space that standard SAEs resolve differently across seeds due to basis ambiguity.

Interpretability Implications: The existence of unstable features suggests that a single SAE dictionary can obscure important cross-seed structure. However, this structure is recoverable.
Methodological Contribution: The authors propose that pooling high-probability features across runs is a viable strategy to construct more stable SAEs without sacrificing reconstruction quality or downstream interpretability performance.
Theoretical Insight: The work clarifies that seed dependence often reflects different basis choices within a shared region of activation space rather than the absence of shared structure. This distinction is crucial for understanding the limits of feature-level interpretability and the potential for recovering identifiable individual features within reproducible subspaces.

The authors remain modest, noting that their low-rank analysis provides evidence for a specific mechanism of seed dependence but does not prove that every unstable feature in real Large Language Models is produced solely by this mechanism. Furthermore, the lack of a stability-EV trade-off is specific to the feature-pool construction method, not necessarily a property of a single SAE training objective.