🤖 machine learning

Ablating Archetypes: The Stability of Archetypal SAEs is an Artifact of Initialization and Metric Design

This paper demonstrates that the reported stability of Archetypal SAEs is an artifact of identical initialization and metric design rather than a genuine convergence property, arguing that true stability must be distinguished from stabilization and verified through trajectory diagnostics and initialization ablations.

Original authors: Michał Brzozowski, Neo Christopher Chung

Published 2026-06-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Michał Brzozowski, Neo Christopher Chung

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Unreliable Map" Problem

Imagine you are trying to understand how a giant, complex machine (like a Large Language Model) works. To do this, researchers use a tool called a Sparse Autoencoder (SAE). Think of an SAE as a translator that breaks the machine's internal thoughts down into a list of simple, understandable "concepts" or "features" (like "politeness," "math," or "coding").

The goal is to find a dictionary of these concepts that is stable. This means if you run the translator twice with slightly different settings, you should get the same list of concepts. If you get a totally different list every time, you can't trust the results.

Recently, a new method called Archetypal SAEs was proposed. The creators claimed this new method was much more stable and reliable than the old way. They said it produced a consistent dictionary of concepts every time.

This paper argues that this claim is an illusion. The authors show that the new method isn't actually more stable; it just looks that way because the researchers cheated the starting line.

Analogy 1: The Race Track (Stability vs. Stabilization)

The paper makes a crucial distinction between two words that sound similar but mean very different things: Stability and Stabilization.

Stabilization (The Real Test): Imagine two runners starting at opposite ends of a track. They run toward the finish line. If they both end up at the exact same spot, we know the track forced them to converge. This proves the path is solid.
Stability (The Fake Test): Imagine two runners who start the race standing right next to each other, holding hands. They take a few steps and stop. They are still standing next to each other. Are they "stable"? Yes. But did they prove the track is good? No. They were just never apart to begin with.

The Paper's Finding:
The "Archetypal SAE" method is like the second runner. The researchers set up the experiment so that both versions of the model started with the exact same dictionary (the same starting point). Because they started identical, they ended identical.

The authors say: "You can't claim you found a better path just because you forced everyone to start at the same place." When they let the Archetypal SAEs start from random, different places (like a real race), they actually performed worse than the old method. They were slower to agree on the final concepts.

Analogy 2: The "Galaxy" Effect (The Measurement Error)

The paper also points out a second problem: how they measured the results. They used a ruler called "Cosine Distance" to see how similar the dictionaries were.

The Problem: Imagine you are looking at a galaxy of stars from Earth. Because the galaxy is so far away, all the stars look like they are in the same tiny direction, even though they are actually millions of miles apart.
In the Paper: The data the models were learning from had a "mean" (an average) that was far away from zero. This made all the "concepts" (stars) cluster tightly in one direction. When the researchers measured the distance between them, the "Galaxy Effect" made them look incredibly similar, even if they were actually different.

The authors fixed this by "centering" the data (subtracting the average), which scattered the stars out. Once they did this, the "Archetypal" method didn't look nearly as stable as before.

The Experiments: Pulling Back the Curtain

To prove their point, the authors ran two specific tests:

The "Remove the Constraint" Test: They took the Archetypal method and turned off its special "convex hull" rule (the rule that forces concepts to stay within a specific shape).
- Result: Surprisingly, the model became more stable (the dictionaries matched better) and actually worked better at reconstructing the data. This proved that the special rule wasn't helping; it was actually getting in the way.
The "Fair Start" Test: They took the old, standard method and gave it the same "head start" (initialization) that the Archetypal method had.
- Result: The old method performed just as well as the Archetypal method. This proved that the "Archetypal" method's success was entirely due to the head start, not the special rule.

The Conclusion

The paper concludes that Archetypal SAEs are not a magic bullet for stability.

The "stability" they reported was an artifact (a side effect) of using the same starting point for every experiment.
When you test them fairly (starting from different points), they don't converge faster; they actually converge slower.
The measurement tool they used was also biased by how the data was prepared.

In short: The new method didn't solve the problem of unreliable maps. It just made it look like the maps were identical because everyone was forced to start from the same spot. To truly know if a method works, you have to let the runners start from different places and see if they still find the same finish line.

Technical Summary: Ablating Archetypes

Problem Statement
Sparse Autoencoders (SAEs) are a standard tool in mechanistic interpretability for decomposing neural network activations into interpretable, sparse features. A critical challenge in this domain is instability: SAEs trained on identical data with different random seeds often learn substantially different dictionaries, making it difficult to distinguish genuine computational features from training artifacts.

Recent work by Fel et al. (2025) proposed Archetypal SAEs, which constrain dictionary atoms to lie within the convex hull of a set of "archetypes" (typically k-means centroids). They reported that this method yields significantly more stable dictionaries at the end of training compared to classical SAEs. This paper argues that the reported stability is not a result of the archetypal constraint itself, but rather an artifact of identical initialization across runs and a preprocessing-dependent metric flaw.

Methodology and Analysis
The authors employ a combination of trajectory analysis, initialization ablations, and metric diagnostics to deconstruct the stability claims of Archetypal SAEs.

Distinction Between Stability and Stabilization:
The paper clarifies a conceptual conflation in the literature:
- Stability (Consistency): Agreement between two independently trained models at the end of training.
- Stabilization: The convergence of two runs starting from genuinely different initializations toward a common solution.
  The authors argue that Archetypal SAEs demonstrate the former trivially (by fixing the start point) but fail to demonstrate the latter.
Trajectory Diagnostics:
Instead of only measuring endpoint similarity, the authors plot the inter-run dictionary distance (mean minimum cosine distance) as a function of training steps.
- Classical TopK (Random Init): Starts with high distance and decreases monotonically, indicating genuine stabilization driven by data constraints.
- Archetypal TopK (Fixed k-means Init): Remains flat at zero distance throughout training because both seeds start from the identical dictionary ( $D=C$ ).
- Archetypal TopK (Random Init): When the archetypal constraint is applied but initialization is randomized, the distance decreases but at roughly half the rate of classical TopK, suggesting the constraint actually impedes convergence in this setting.
The "Galaxy-Far-Away" Effect (Metric Pathology):
The paper identifies a geometric confound in using cosine distance on uncentered activations. In LLM residual streams, activations have a non-negligible mean ( $\mu$ ). k-means centroids on uncentered data cluster in a narrow cone around $\mu$ . Consequently, the cosine distance between any two atoms is dominated by the mean offset rather than their directional differences, artificially inflating similarity scores. The authors demonstrate that centering the activations (subtracting the mean) eliminates this bias, causing pre-centering cosine similarities to collapse from ~0.36 to ~0.00.
Ablation Studies:
Using a controlled setup on DinoV2 (vision) and Pythia-160m (language) data, the authors perform two ablations:
- Disabling the Convex-Hull Projection: Removing the projection constraint from Archetypal SAEs (while keeping k-means initialization) results in higher stability (lower Hungarian distance) than the full Archetypal SAE, while improving reconstruction ( $R^2$ ).
- Fixing Classical SAE Initialization: Initializing a standard TopK SAE with the same k-means centroids used in Archetypal SAEs (without the convex-hull constraint) yields stability metrics identical to the ablated Archetypal SAE.

Key Results

Initialization is the Driver: The stability reported by Fel et al. (2025) is primarily due to the deterministic k-means initialization shared across seeds, not the archetypal constraint. When initialization is randomized, the archetypal constraint slows down stabilization compared to classical SAEs.
Reconstruction Cost: The convex-hull projection constraint in Archetypal SAEs reduces reconstruction quality ( $R^2$ ) without providing a stabilization benefit in the tested settings.
Metric Validity: The standard cosine stability metric is invalid for uncentered activations due to the "galaxy-far-away" effect. Centering is a necessary preprocessing step to ensure stability metrics reflect genuine directional differences.
Quantitative Evidence: In experiments, a Classical SAE with fixed k-means initialization achieved a Hungarian distance of 0.068 (Vision) and 0.112 (Pythia), matching the ablated Archetypal SAE and significantly outperforming the full Archetypal SAE (0.243 and 0.170, respectively).

Significance and Claims
The paper does not claim that Archetypal SAEs are useless for all purposes (e.g., visualization or bounded norms), but it rigorously refutes the claim that they offer superior stabilization for feature discovery.

Methodological Correction: The authors assert that claims of SAE stability must be validated through trajectory diagnostics (showing convergence from different starts) rather than just endpoint agreement.
Reproducibility Warning: They highlight that the "stability" of Archetypal SAEs is an artifact of an experimental design that fails to isolate the constraint from its initialization.
Interpretability Implications: If stability is used as a proxy for feature validity (i.e., "if it appears across seeds, it is real"), then methods that are stable by construction (fixed initialization) provide no such evidence. True evidence requires convergence from diverse starting points.

The study concludes that while the archetypal construction has geometric virtues, it does not constitute a viable route to stabilization in the current setting, and that the field must distinguish between stability (agreement) and stabilization (convergence) to avoid misleading interpretations of mechanistic findings.