SAEVerbalizer: Generating Explanations for Sparse Autoencoder Features via Representation Verbalization

Technical Summary: SAEVerbalizer

Problem Statement
Sparse Autoencoders (SAEs) are widely used to extract interpretable features from the dense internal representations of Large Language Models (LLMs) by mapping them into a high-dimensional sparse space. However, a critical bottleneck remains: explaining what these extracted features represent in natural language. Current state-of-the-art methods rely on bottom-up interpretation, which involves running an LLM over a large corpus to identify top-activating examples for a feature and then prompting an external LLM to summarize the shared pattern.

The authors identify two primary limitations in this existing paradigm:

1. Superficial Explanations: By inferring meaning solely from activation examples, these methods describe what the examples have in common rather than directly accessing the semantic content encoded within the feature's representation itself.
2. Computational Inefficiency: Generating explanations for millions of features requires repeated corpus-scale inference, activation computation, and example retrieval, which is not scalable.

Methodology: SAEVerbalizer
The paper introduces SAEVerbalizer, a framework that shifts the explanation process from observing external behavior to processing internal representations directly. Instead of retrieving activation examples, SAEVerbalizer treats the SAE decoder direction as the sole input to generate an explanation.

• Core Mechanism: The framework injects an SAE decoder direction (a vector representing a specific feature) directly into the token representations of an LLM at a designated injection layer ($L_{inj}$). The LLM's downstream layers are then fine-tuned to generate a natural language explanation based on this injected representation.
• Injection Strategy: The decoder direction is added to the token representations in a fixed injection span using a norm-matched additive injection:
  $$h'_{b,s} = h_{b,s} + \alpha \bar{n}_b \hat{v}_{f_b}$$
  where $\hat{v}$ is the normalized direction, $\bar{n}_b$ is the mean norm of the pre-injection representations, and $\alpha$ controls the injection strength.
• Training:
  • Verbalizer Fine-tuning: The LLM is fine-tuned on high-quality feature–explanation pairs (sourced from Neuronpedia and filtered for coherence and specificity). Only the layers downstream of the injection layer are updated, preserving the representation space where the SAE directions are defined.
  • Adapter for Cross-LLM Transfer: To apply a verbalizer trained on one LLM to SAE features from a different LLM, a lightweight affine adapter is trained. This adapter maps decoder directions from the source LLM's representation space into the target verbalizer's injection-layer space, trained on aligned hidden representations from unlabeled text.
• Inference: Once trained, the verbalizer can explain unseen features by injecting the corresponding decoder direction and generating text autoregressively, without needing to retrieve or process activation examples.

Key Contributions

1. A New Paradigm for Interpretation: The authors propose a lightweight approach to explain SAE features directly from decoder directions, bypassing the need for corpus-scale inference and example retrieval.
2. Trainable Capability: They demonstrate that the ability to verbalize internal representations is a trainable capability that generalizes to unseen features and transfers across separately trained SAE dictionaries within the same representation space.
3. Cross-LLM Scalability: Through lightweight adapters, the framework extends verbalization capabilities to SAE features from different LLMs without requiring source-specific supervision or retraining the verbalizer backbone.
4. Intervention Analysis: The paper provides empirical evidence that the verbalizer captures compositional semantics (when multiple directions are injected) and signed relationships (when directions are reversed), suggesting it learns the underlying geometry of the feature space rather than just memorizing associations.

Experimental Results
The authors evaluated SAEVerbalizer using Gemma LLMs (1B, 4B, 27B) and Gemma Scope 2 SAEs.

• Performance: The verbalizer achieves significant Reference Agreement (RA) scores (e.g., 52.3% on the Global Train-Standard set for the 27B model with 48k training pairs), demonstrating that it can generate explanations consistent with high-quality human-curated references.
• Generalization: The model generalizes to unseen features and transfers directly to a different SAE dictionary (width-65k) trained on the same LLM without additional fine-tuning.
• Transferability: When using adapters to transfer from 1B and 4B models to a 27B verbalizer, the system successfully leverages the stronger verbalization capability of the larger model, improving performance over native verbalizers trained on smaller models.
• Robustness: The system is robust to variations in prompt design and injection strength, provided the injection is sufficiently strong.
• Qualitative Findings:
  • Compositionality: Joint injection of two directions yields explanations that combine the meanings of both features.
  • Direction Reversal: Reversing the sign of a decoder direction produces semantically related but shifted explanations (e.g., "lessons learned" vs. "lesson plan"), indicating the verbalizer captures the directional nature of the feature space.

Significance and Claims
The paper claims that SAEVerbalizer offers a more direct and computationally efficient route from learned representations to feature explanations. By grounding explanations in the LLM's internal representation space rather than inferring them from activation texts, the method addresses the limitations of superficiality and inefficiency in current automated interpretation.

The authors position SAEVerbalizer not as a replacement for all interpretability methods, but as a trainable and partially reusable complement to existing approaches. It enables the generation of explanations for unseen features and across different model architectures with minimal overhead, facilitating the scaling of SAE interpretation to millions of features. The work suggests that internal representation verbalization is a learnable capability that can be transferred and adapted, opening new avenues for analyzing the semantic structure of LLMs.

Limitations
The authors acknowledge several limitations:

• Coverage: Experiments are currently limited to Gemma LLMs and Gemma Scope 2 SAEs; generalization to other families is unverified.
• Validation: Evaluation relies on Reference Agreement with filtered Neuronpedia explanations, which serve as a proxy for quality but do not constitute absolute ground truth.
• Supervision: The current pipeline relies on computationally expensive filtering of Neuronpedia data to create training pairs, though the authors note alternative supervision sources could be used.
• Run Variability: Results are reported from single runs, without measuring variability across random seeds.