Steering LLMs? Actually, Sparse Autoencoders can outperform simple baselines

Technical Summary: Steering LLMs with Sparse Autoencoders

Problem Statement
Sparse Autoencoders (SAEs) have been proposed as a tool for interpreting Large Language Models (LLMs) by discovering monosemantic features within the residual stream, based on the Linear Representation Hypothesis and the superposition hypothesis. While SAE features are theoretically capable of controlling model generation (steering), recent benchmarks such as AxBench (Wu et al., 2025) suggested that SAEs underperform significantly compared to simple baselines like prompting. Wu et al. (2025) concluded that "Even simple baselines outperform Sparse Autoencoders" for steering tasks. Furthermore, previous work (Wang et al., 2025) suggested that high interpretability and steering utility are strongly correlated with high sparsity (low $\ell_0$), a condition often met by features annotated in repositories like Neuronpedia. The authors of this paper argue that the poor performance reported in Wu et al. (2025) may stem from suboptimal feature labeling and selection strategies rather than an inherent limitation of SAEs.

Methodology
The paper proposes a supervised pipeline for labeling and selecting SAE features to improve steering performance, contrasting with previous unsupervised or LLM-based labeling approaches.

1. Supervised Labeling Pipeline:

   • Data Source: The authors utilize Stack Exchange datasets (e.g., academia, biology, sports) where user-assigned tags serve as ground-truth multi-labels.
   • Feature Probes: For each SAE feature, a linear probe is constructed by calculating the feature's activation frequency over a dataset. A binary prediction is made based on whether the frequency exceeds a threshold $\tau$.
   • Calibrated F1 Scoring: To match features to labels, the authors compute a "calibrated F1" score ($F1^c$). This metric adjusts the standard F1 score to account for label support (frequency), preventing bias toward rare or common labels. The optimal threshold $\tau$ is selected to maximize this score.
   • Denoising and Selection: To ensure high-quality matches, the pipeline applies three criteria:
     1. Label Support Threshold: Filters out labels with fewer than 50 occurrences to reduce variance.
     2. Output Score Proxy: Uses a threshold ($10^{-3}$) based on the output score from Arad et al. (2025) to filter for features with inherent steering potential.
     3. Top-K Selection: Retains the top 50 feature-label pairs per category.

2. Steering Mechanism:

   • The authors employ the activation editing approach of Templeton et al. (2024). During inference, specific feature activations ($z_f$) are clamped to a value $\alpha$. The reconstruction error is preserved, and the edited latent vector is passed through the decoder.
   • Evaluation: The method is evaluated on AxBench, which rates generated text on fluency, instruction following, and concept incorporation. The authors optimize the steering strength $\alpha$ individually for each feature-label pair using cross-validation, rather than using a single global value.

3. Experimental Setup:

   • Models: Experiments were conducted on Gemma 2 (9B) using Gemma Scope SAEs (width 131k) inserted into the residual stream at layers 17 and 32.
   • Baselines: Comparisons include the "NP random" baseline (randomly selected features with Neuronpedia labels) and the prompt steering baseline from Wu et al. (2025).

Key Results

• Performance Parity with LoRA: When features are selected and labeled via the proposed supervised pipeline, SAEs achieve steering performance on the AxBench benchmark that is close to on par with LoRA (Low-Rank Adaptation) baselines. This significantly outperforms the results reported by Wu et al. (2025).
• Causal Validity: The pipeline selects features that are surprisingly causal of their identified labels. Even when the "output score" component (a proxy for steering ability) is removed from the selection process, the remaining features selected by the supervised labeling still demonstrate superior steering performance compared to random Neuronpedia selections.
• Sparsity Independence: Contrary to findings in Wang et al. (2025), the authors find that high sparsity (low $\ell_0$) is not strictly crucial for successful steering. Both low- and high-sparsity SAEs in the Gemma Scope model were effective for steering tasks.
• Prompting Baseline: While the proposed SAE method improves significantly over previous SAE attempts, the authors note that simple prompting remains a strong baseline, likely because LLMs are explicitly trained to follow instructions.

Significance and Claims
The paper serves as a partial rebuttal to the claim that SAEs are ineffective for steering. The authors argue that the negative results in Wu et al. (2025) did not do SAEs "full justice" due to the reliance on pre-existing, potentially suboptimal labels and a lack of supervised feature selection.

The primary significance lies in demonstrating that:

1. Labeling Matters: The method of labeling SAE features is critical. A supervised approach using calibrated metrics and support thresholds yields features with genuine causal impact on model generation.
2. Robustness of SAEs: SAEs can be competitive with parameter-efficient fine-tuning methods (LoRA) for steering, validating their utility as an interpretability tool.
3. Causal Ground Truth: In a research setting, successful feature steering confirms the "ground truth" of a feature's label, distinguishing it from artifacts of dataset idiosyncrasies (polysemanticity) that might mislead correlation-based interpretability.

The authors maintain a modest tone, acknowledging that while SAEs are powerful for research and understanding model internals, prompting remains a difficult baseline to beat for practical applications due to the nature of LLM training. They also note that feature steering involves "controlled damage" to the model, making it less suitable for production environments compared to standard fine-tuning.