Technical Summary: On the Relationship Between Activation Outliers and Feature Death in Sparse Autoencoders

1. Problem Statement

Sparse Autoencoders (SAEs) are designed to decompose neural network activations into interpretable, sparse features to reverse the phenomenon of superposition (where networks encode more concepts than dimensions by overlapping directions). However, a critical failure mode known as feature death occurs when learned dictionary features never activate on any input. This wastes dictionary capacity and forces surviving features to compensate, potentially reintroducing superposition.

The paper identifies a significant inconsistency in SAE performance: identical SAE configurations yield near-zero dead features on models like GPT-2 but over 70% dead features on models like AlphaFold3. Existing revival techniques (e.g., AuxK, ghost gradients, resampling) treat feature death as a training dynamics problem, attempting to inject gradient signals into stuck features. However, these methods fail to resolve death rates in the most severe cases, suggesting the root cause lies not in training dynamics but in the initialization state of the activations.

2. Methodology and Theoretical Framework

2.1 Dimension-Level Activation Outliers

The authors propose that dimension-level activation outliers are the primary cause of feature death. Unlike token-level outliers (spikes in specific inputs), these are dimensions where the mean activation magnitude ($\mu$) is large relative to the per-token standard deviation ($\sigma$). This creates a near-constant offset across all inputs.

The authors formalize outlier severity as the ratio:
$$\gamma = \frac{\|\mu\|}{\|\sigma\|}$$
where $\mu$ is the mean activation vector and $\sigma$ is the per-token standard deviation vector.

2.2 Geometric Mechanism of Death

At initialization (with zero biases), the pre-activation for feature $i$ is $z_i = w_i \cdot x$. Decomposing the input $x$ into its mean and variation ($x = \mu + (x - \mu)$), the pre-activation becomes:
$$z_i = \underbrace{w_i \cdot \mu}_{\text{constant shift}} + \underbrace{w_i \cdot (x - \mu)}_{\text{input-dependent signal}}$$

When $\gamma$ is high, the constant shift dominates the signal. This leads to two distinct failure modes at initialization:

1. Dead-by-ReLU: Features with weights anti-aligned with $\mu$ ($w_i \cdot \mu \ll 0$) receive permanently negative pre-activations. Since ReLU outputs zero for negative values, these features never fire and receive no gradient.
2. Dead-by-TopK: In TopK SAEs, features with moderate positive shifts may pass ReLU but lose the competition to features with large positive shifts. Only the $k$ features most aligned with $\mu$ win the TopK selection, leaving the rest dead.

2.3 Analytical Derivation

The paper derives closed-form expressions for the probability of feature death based solely on $\gamma$:

• Dead-by-ReLU: $P(\text{dead}) = \Phi\left(\frac{-C}{\gamma}\right)$, where $\Phi$ is the standard normal CDF and $C$ depends on the number of evaluation samples.
• Dead-by-TopK: $P(\text{dead}) \approx \Phi\left(\frac{t_k - C}{\gamma}\right)$, where $t_k$ is the quantile corresponding to the TopK threshold.

These formulas predict that as $\gamma$ increases, the death rate monotonically increases, approaching 50% for ReLU and $1 - k/n$ for TopK at extreme severities.

2.4 Experimental Setup

The authors validate these claims across 454 model-layer combinations spanning language, vision, protein, and genomic models. They:

• Compute $\gamma$ on real activations (using per-token LayerNorm).
• Train SAEs with and without auxiliary losses (AuxK).
• Conduct synthetic experiments where $\gamma$ is controlled to isolate causality.
• Test mean-centering (subtracting $\mu$ from inputs or initializing the bias to $\mu$) as a mitigation strategy.

3. Key Contributions

1. Outlier Severity Diagnostic ($\gamma$): The authors derive $\gamma$ from first principles, showing it analytically determines the fraction of features dead at initialization. This diagnostic correlates strongly with observed death rates across diverse models (Spearman $\rho = 0.89$ for TopK, $\rho = 0.82$ for ReLU) without any fitting.
2. Geometric Cause of Death: The paper establishes that feature death in high-$\gamma$ models is a geometric initialization problem, not a training dynamics issue. The activation mean creates input-independent pre-activation shifts that predetermine feature fate before training begins.
3. Recovery Dynamics Analysis: The authors characterize two distinct revival pathways:
   • Fast: Dead-by-TopK features revive as competing features shrink during training.
   • Slow: Dead-by-ReLU features revive only as the SAE bias slowly learns to absorb the mean offset $\mu$. This process is prohibitively slow at high $\gamma$, explaining why training alone often fails to recover features.
4. Validation via Mean-Centering: The paper demonstrates that initializing the SAE bias to the activation mean (effectively mean-centering) eliminates outlier-induced death at initialization. This confirms the geometric mechanism and provides a principled rationale for when this preprocessing is necessary.

4. Results

• Correlation: $\gamma$ accurately predicts initial death rates across 454 model-layer combinations. Synthetic experiments confirm a perfect rank correlation ($\rho=1.0$) between increasing $\gamma$ and increasing death rates.
• Training Limitations: In high-$\gamma$ scenarios (e.g., $\gamma \ge 30$), standard training fails to recover features even after 2 million steps. The bias learning process is the bottleneck; features with permanently negative pre-activations remain dead because the bias cannot learn the large offset quickly enough.
• AuxK Limitations: AuxK accelerates the recovery of Dead-by-TopK features by providing gradient signals, but it does not speed up the bias learning required for Dead-by-ReLU recovery. Consequently, AuxK helps at moderate $\gamma$ but fails to resolve death at high $\gamma$.
• Mean-Centering Efficacy:
  • On synthetic data ($\gamma=40$), mean-centering reduces dead features to near zero and lowers reconstruction error by an order of magnitude.
  • On real models (e.g., ESM3, AlphaFold3), mean-centering reduces initial death from >80% to near zero.
  • Feature Quality: Mean-centered SAEs produce more monosemantic features. On ESM3, a mean-centered SAE with 2,048 features recovers more biological concepts (100) than a baseline SAE with 8,192 features (73).
• Residual Death: Mean-centering does not eliminate all death. In extremely low-rank layers (e.g., Evo1), where variance is concentrated in very few directions, features still die due to TopK competition. The authors suggest PCA whitening or Active Subspace Initialization as remedies for these specific cases.

5. Significance and Claims

The paper claims to provide a principled basis for understanding and mitigating feature death in SAEs, shifting the paradigm from "fixing training dynamics" to "correcting initialization geometry."

• Practical Guidance: The authors propose a simple diagnostic workflow: compute $\gamma$ for a layer. If $\gamma$ is high, mean-centering (initializing bias to the activation mean) is necessary to prevent feature death. If $\gamma$ is low, standard training suffices.
• Efficiency: Mean-centering eliminates the need for computationally expensive revival techniques (like AuxK) in high-$\gamma$ regimes and allows smaller dictionaries to achieve the same or better interpretability than larger, uncentered ones.
• Theoretical Insight: The work clarifies that the variability in SAE performance across different domains (language vs. biology) is driven by the activation geometry of the source models, not the SAE architecture itself.

The paper concludes that for most models, initializing the SAE bias to the geometric median of activations is a zero-cost, high-impact step that ensures the effective dictionary size matches the intended size, thereby enabling more reliable mechanistic interpretability across diverse model families.