Technical Summary: When Pruning Meets Interpretability

Problem Statement

Sparse autoencoders (SAEs) have become a standard tool for mechanistic interpretability in large language models (LLMs), enabling the recovery of monosemantic features from the superposition of neuron activations. However, a critical gap exists in understanding how these interpretability tools behave when the underlying models undergo post-hoc compression via weight pruning.

While pruning methods (ranging from classical magnitude-based approaches to modern activation-aware techniques) are widely used to reduce deployment costs, they alter the model's weight matrices and, consequently, the distribution of internal activations. SAEs are trained on the activation distribution of a specific dense model; when this distribution shifts due to pruning, the SAE's latents may fail silently. Crucially, standard language modeling benchmarks often fail to detect these failures, leaving organizations without a theoretical framework to determine if an existing SAE remains valid after compression, or why different pruning methods cause varying degrees of degradation.

Methodology

The authors propose a unified perturbation-theoretic framework to analyze the interaction between pruning and SAEs.

1. Theoretical Framework: Perturbation Energy
The paper models pruning as an activation perturbation. When a weight matrix $W$ is modified to $W' = W + \Delta W$, the resulting activation shift is $\delta = \Delta W x_{in}$. The authors define perturbation energy ($\varepsilon^2$) as a covariance-weighted norm:
$$\varepsilon^2 = \mathbb{E}[\|\delta\|^2] = \text{tr}(\Delta W \Sigma_{x_{in}} \Delta W^\top)$$
where $\Sigma_{x_{in}}$ is the second-moment matrix of layer inputs.

Using the Lipschitz continuity of the SAE, the authors derive a bound on expected reconstruction degradation. This bound demonstrates that degradation is driven by the perturbation energy $\varepsilon^2$ and the SAE's Lipschitz constant. The theory posits that pruning methods differ fundamentally in how they approximate the minimization of $\varepsilon^2$:

• MAGNITUDE Pruning: Minimizes the Frobenius norm of weight changes ($\|\Delta W\|_F^2$), effectively assuming an identity covariance matrix ($\Sigma_{x_{in}} \propto I$). It ignores activation geometry.
• WANDA: Minimizes a diagonal approximation of the perturbation energy, accounting for input activation statistics.
• SPARSEGPT: Uses second-order Hessian information to approximately minimize the full perturbation energy $\varepsilon^2$.

2. Evaluation Suite
To validate the theory, the authors utilize the SAEBench evaluation suite, arguing that four specific metric categories are jointly sufficient to capture all SAE failure modes:

• Core Metrics: Measure aggregate reconstruction fidelity (e.g., KL divergence, MSE).
• Feature Absorption: Detects if individual features break or if unrelated features absorb their information (false negatives/positives).
• Spurious Correlation Removal (SCR): Tests if latent ablations retain their intended causal effects.
• Targeted Probe Perturbation (TPP): Verifies that interventions remain selective and do not degrade unrelated concepts.

3. Experimental Setup
Experiments were conducted on four model architectures (Pythia-70M, Gemma-2-2B, Gemma-2-9B, Mistral-7B) using pre-trained SAEs from SAELens. The SAE weights were held fixed to isolate the effects of pruning. Three pruning methods were applied post-hoc at 25%, 40%, and 50% sparsity. The study also explored a layer-wise sparsity allocation strategy based on observed sensitivity patterns.

Key Contributions and Results

1. Theoretical Separation of Pruning Methods
The study provides a formal explanation for the empirical performance of pruning methods. It establishes a hierarchy of approximation fidelity regarding perturbation energy:
$$\text{MAGNITUDE} \prec \text{WANDA} \prec \text{SPARSEGPT}$$
Results confirm that MAGNITUDE pruning, by ignoring activation geometry, induces the highest perturbation energy and causes the most severe SAE degradation. In contrast, activation-aware methods (WANDA and SPARSEGPT) implicitly control perturbation energy, preserving SAE behavior substantially better.

2. Limitations of Aggregate Metrics
The experiments reveal that "Core" metrics (aggregate reconstruction) can be misleading. For instance, MAGNITUDE pruning on middle layers of Gemma-2-2B showed superficially reasonable KL divergence scores ($\ge 0.96$), yet the SCR and TPP scores collapsed (degrading by up to 80%). This confirms that aggregate fidelity is insufficient to certify SAE reliability; feature-level and causal metrics are necessary to detect silent failures.

3. Structural Vulnerability of Middle Layers
A consistent structural vulnerability was identified across all models and pruning methods: middle layers (layers 9–17 in Gemma-2-2B) are significantly more sensitive to pruning than early or late layers.

• Middle layers exhibited the highest degradation in all metrics, even when local perturbation energy was low.
• The authors hypothesize this is due to the asymmetric propagation of representational damage: middle layers act as primary recipients of perturbation energy from upstream weights, whereas late layers operate on fully composed representations with less upstream dependency.

4. Layer-wise Sparsity Allocation
Guided by the sensitivity profile, the authors propose a non-uniform sparsity schedule: assigning lower sparsity to early layers (to preserve foundational features) and higher sparsity to late layers.

• On Gemma-2-2B, this strategy achieved lower perplexity compared to uniform sparsity at the same average compression level.
• This suggests that sensitivity profiles can serve as a practical signal for compression scheduling, though the authors note this is a preliminary result rather than a fully optimized allocation method.

5. Cross-Architecture Generalization
The theoretical ordering (SPARSEGPT $\succ$ WANDA $\succ$ MAGNITUDE) and the middle-layer sensitivity pattern held across all four tested architectures, including smaller models (Pythia-70M) where MAGNITUDE pruning caused a catastrophic collapse in SAE performance (KL score dropping from 0.926 to 0.642).

Significance and Claims

The paper claims to provide the first theoretical framework characterizing when pruning preserves SAE validity without retraining. Its significance lies in:

• Bridging the Gap: It connects the fields of model compression and mechanistic interpretability, showing that the choice of pruning method is critical for preserving interpretability tools.
• Diagnostic Utility: It demonstrates that standard performance benchmarks are insufficient for validating compressed models for interpretability; specific SAE-centric metrics (Feature Absorption, SCR, TPP) are required.
• Practical Guidance: It offers a theoretical justification for using activation-aware pruning methods (WANDA, SPARSEGPT) over magnitude-based ones when interpretability is a priority.
• Structural Insight: It identifies a previously uncharacterized structural vulnerability in middle layers, suggesting that uniform compression strategies may be suboptimal for preserving internal representations.

The authors remain modest regarding their findings, noting that their layer-wise allocation is a preliminary illustration and that their theoretical framework currently explains the ordering of methods but does not yet fully explain the theoretical origin of the middle-layer sensitivity. Future work is needed to extend these findings to structured pruning, quantization, and different model families.