Technical Summary: Evaluating Adversarial Robustness of Concept Representations in Sparse Autoencoders

Problem Statement

Sparse Autoencoders (SAEs) have emerged as a primary tool for mechanistic interpretability, mapping the dense internal activations of Large Language Models (LLMs) to sparse, human-interpretable concept representations. While existing evaluations of SAEs focus on reconstruction-sparsity tradeoffs, human interpretability, and feature disentanglement, they largely overlook a critical dimension: robustness to input perturbations.

Current evaluations often assume static, coherent, and unperturbed inputs. However, in adversarial contexts—where an explanation provider and a consumer may have conflicting goals—SAE representations could be vulnerable. If an adversary can craft minimal input perturbations that manipulate SAE activations without significantly altering the base LLM's behavior, the fidelity of concept labeling is compromised. This raises concerns about the reliability of SAEs for model monitoring, safety oversight, and downstream applications that depend on stable concept-level representations.

Methodology

Theoretical Framework

The authors formalize SAE robustness as input-space optimization problems. They posit a "ground-truth" concept mapping $f_c$ from the input space $X$ to a semantic concept space $C$. Robustness is defined by the preservation of the alignment between the learned sparse latent space $Z$ and $C$ under adversarial perturbations.

The core assumption is that the ground-truth mapping is bi-Lipschitz, meaning small changes in input token sequences (measured by Levenshtein distance) correspond proportionally to small changes in semantic concepts. This allows the search for adversarial examples to be conducted directly in the input space $X$ rather than the hypothetical concept space $C$.

The evaluation framework defines robustness through two optimization objectives:

1. Untargeted: Maximize the distance between the original SAE activation $z_1$ and the perturbed activation $z'_1$ given a bounded input edit distance.
2. Targeted: Minimize the distance between the perturbed activation $z'_1$ and a target activation $z_2$ (from a semantically unrelated input) while keeping the input perturbation bounded.

Evaluation Dimensions

The authors propose a structured framework combining three binary dimensions to define eight distinct adversarial scenarios:

1. Semantic Goal: Untargeted (altering SAE activations generally) vs. Targeted (steering activations toward a specific unrelated input).
2. Activation Goal:
   • Population Level: Manipulating the set of top-$k$ activated latents, measured by concept overlap ratio.
   • Individual Level: Activating or deactivating a specific SAE feature, measured by binary rank changes.
3. Perturbation Mode: Minimal input changes, specifically replacing a single token or appending a single-token suffix.

Attack Algorithm

To solve these optimization problems, the authors adapt the Greedy Coordinate Gradient (GCG) algorithm, originally designed for prompt optimization in LLMs. Since the distance metrics in the sparse latent space are non-differentiable, the method employs a hybrid approach:

• Gradient Computation: Gradients are computed using differentiable loss functions (e.g., cosine similarity, log-likelihood) over the continuous SAE representation space.
• Candidate Selection: Adversarial candidates are generated and evaluated using the original non-differentiable distance metrics (e.g., concept overlap ratio, binary activation status) in the discrete input space.

Key Contributions

1. Identification of a Critical Gap: The paper identifies robustness as a fundamental, yet underexplored, dimension in SAE evaluation, arguing that concept representations must be stable under input perturbations to be considered reliable.
2. Comprehensive Evaluation Framework: The authors introduce a theoretically grounded framework that defines SAE robustness across semantic and activation goals, enabling systematic testing of adversarial vulnerabilities.
3. Generalized Attack Methodology: They propose a generalized input-level attack algorithm adapted from GCG, capable of generating adversarial inputs that manipulate SAE interpretations.
4. Empirical Evidence of Fragility: Extensive experiments across multiple open-source LLMs (Gemma2, Llama3), SAE architectures (JumpReLU, Top-K), and datasets demonstrate that SAE concept representations are highly fragile.

Experimental Results

The authors evaluated their framework on Gemma2-2B, Gemma2-9B, and Llama3-8B with various SAE widths and layers, using datasets including AdvBench, AG News, SST2, and a generated "Art & Science" dataset.

• High Success Rates: Tiny adversarial perturbations (often a single token replacement) effectively manipulate concept-based interpretations.
  • Population Level: Untargeted attacks successfully reduced the concept overlap between the original and perturbed inputs to ~20–30%, while targeted attacks increased the overlap with an unrelated target to ~50–60%.
  • Individual Level: Over 60% of selected SAE latents were manipulable in targeted settings. In deactivation scenarios, success rates reached approximately 90%.
• Decoupling from LLM Robustness: Crucially, linear probes trained on the base LLM's hidden states maintained high accuracy (>90%) in detecting semantic concepts (e.g., harm, sentiment, topic) even after the adversarial perturbations. This indicates that the LLM's internal semantic representations remain stable, while the SAE's mapping to concepts is the point of failure.
• Non-Adversarial Vulnerability: Even non-adversarial perturbations, such as replacing a word with a synonym, reduced concept overlap by roughly 30%, suggesting inherent fragility beyond malicious attacks.
• Transferability: Attacks transferred across different models (e.g., from Gemma2 to Llama3) showed performance degradation but remained effective, indicating a general vulnerability in SAE-based interpretation.

Significance and Claims

The paper concludes that SAE concept representations are fragile and, without further denoising or post-processing, may be ill-suited for applications requiring high reliability, such as model monitoring and safety oversight.

• Implications for Interpretability: The findings challenge the assumption that SAE latents provide a stable, monosemantic view of model internals. If an adversary can easily alter the "concept" an SAE reports without changing the model's actual output or semantic understanding, the trustworthiness of these explanations is compromised.
• Future Directions: The authors argue that robustness must be a core component of the SAE evaluation agenda. They suggest that future work should focus on developing methods to denoise SAE activations and isolate reliable signals. Furthermore, they posit that these vulnerabilities likely extend to other SAE-inspired approaches, such as transcoders and crosscoders, highlighting the need for robustness-aware objectives in mechanistic interpretability research.

The paper does not claim that SAEs are useless, but rather that their current deployment in adversarial or high-stakes environments is risky without addressing these stability issues. The proposed framework serves as a diagnostic tool to identify and filter out non-robust interpretations.