Beyond I'm Sorry, I Can't: Dissecting Large Language Model Refusal

1. Problem Statement

Large Language Models (LLMs) aligned for safety rely heavily on refusal behavior (e.g., responding "I'm sorry, I can't help with that") to reject harmful prompts. However, the internal mechanisms driving this behavior are poorly understood. Current safety relies on behavioral training (RLHF, SFT) and post-hoc filters, which are often brittle, prone to "reward hacking," and fail to explain why a model refuses or complies.

• The Gap: Existing mechanistic interpretability tools (like linear probes) assume features contribute independently and linearly. They fail to capture non-linear interactions and redundancy (the "Hydra effect"), where multiple features implement the same function, and suppressing one causes another to activate to preserve the behavior.
• Goal: To move beyond trial-and-error alignment by tracing refusal back to specific, causal Sparse Autoencoder (SAE) features and understanding their complex interactions to enable fine-grained auditing and targeted intervention.

2. Methodology

The authors propose an end-to-end pipeline to identify causal features governing refusal in two instruction-tuned models: Gemma-2-2B-IT and LLaMA-3.1-8B-IT. They utilize Sparse Autoencoders (SAEs) trained on residual-stream activations to disentangle superposed vectors into interpretable latent features.

The pipeline consists of three distinct stages:

Stage 1: Refusal Mediating Direction & Candidate Selection

• Steering Vector: Following Arditi et al. (2024), they compute a "refusal steering vector" by taking the difference between residual-stream activations of harmful prompts (refusal) and matched benign prompts.
• Feature Selection: They calculate the cosine similarity between this steering vector and the decoder weights of SAE features. They select the Top-K features most aligned with the refusal direction across layers (excluding early layers 1-2 which encode grammar) as the initial candidate set.

Stage 2: Greedy Filtering (Minimal Faithful Set)

• Objective: Prune the Top-K set to find the minimal subset of features whose ablation (zeroing out) flips the model from refusal to compliance (jailbreak).
• Algorithm: They use a block-wise selection algorithm (Algorithm 1). Iteratively, they sample blocks of features and ablate them. They measure the impact on the probability of the model emitting the token "I" (a strong proxy for refusal).
• Faithfulness Check: If ablating a block significantly changes the "I" token probability, the block is kept; otherwise, it is discarded. This results in a "minimal faithful set" of causal features.

Stage 3: Interaction Discovery (Factorization Machines)

• The Hydra Effect: Initial tests revealed that "critical" features identified in Stage 2 were sometimes inactive during jailbreaks. Removing these inactive features caused the jailbreak to fail, implying that ablating active features triggered dormant "redundant" features to compensate.
• Factorization Machine (FM): To capture these non-linear dependencies, the authors train a second-order Factorization Machine on the SAE activations.
  • The FM models not just linear weights but also pairwise feature interactions using low-dimensional latent embeddings.
  • They rank newly discovered features based on the cosine similarity of their FM embeddings to the Stage 2 features.
  • These additional features are added to the ablation set to fully map the "causal neighborhood."

3. Key Contributions

1. First End-to-End Pipeline: A novel framework combining SAEs, greedy pruning, and Factorization Machines to isolate causal refusal features.
2. Non-Linear Interaction Modeling: Introduction of FMs to recover features that are causally linked via non-linear interactions, which linear probes miss.
3. Discovery of Redundancy: Empirical evidence of the "Hydra effect" in safety mechanisms, where redundant features remain dormant until primary features are suppressed, then activate to maintain refusal behavior.
4. Open Resources: Release of code and specific feature indices to facilitate future research on interpretable and safe LLMs.

4. Key Results

• Jailbreak Success Rates (ASR):
  • LLaMA-3.1-8B: Stage 2 ablation achieved 71% ASR. Stage 3 (adding FM-discovered features) increased unique features but slightly adjusted ASR to 57% (due to removing redundant features that were non-trivially contributing).
  • Gemma-2-2B: Stage 2 achieved 4% ASR (due to model size/difficulty), but Stage 3 increased unique features significantly, achieving 31% ASR.
• Feature Semantics:
  • Features were mapped to the Coconot unsafe taxonomy.
  • While some features encoded harm/violence, a majority were associated with programming constructs, punctuation, and legal/regulatory content.
  • Human Study: A human annotation study comparing activation-based labels to human interpretation showed only 13.42% accuracy, suggesting current SAE features are not yet perfectly disentangled for human intuition, though they are functionally causal.
• Redundancy Analysis:
  • In LLaMA, 74% of redundant features activated on system tokens (specifically <|begin of text|>) only after the primary active set was ablated.
  • Steering Experiments: Clamping these redundant features to high positive values on benign prompts induced refusal or harmful-sounding responses (e.g., "Meditation is illegal"), confirming their causal role in safety behavior.
• Comparison with Linear Probes: A linear probe approach identified features that resulted in significantly fewer jailbreaks (only 101 samples vs. FM-based results), proving that refusal mechanisms rely heavily on non-linear feature interactions.

5. Significance and Future Work

• Mechanistic Understanding: This work shifts the paradigm from viewing refusal as a "black box" behavioral output to understanding it as a sparse sub-circuit involving complex, non-linear interactions and redundancy.
• Targeted Intervention: By identifying the specific "hydra-heads" (redundant features), safety engineers can design more robust interventions that suppress the entire network of refusal features, rather than just the most obvious ones.
• Future Directions:
  • Characterizing the safety benefits/risks of redundancy.
  • Analyzing how different jailbreak strategies (role-play, multi-turn) perturb activation spaces.
  • Hardening refusal for specific reasoning types (e.g., legal reasoning).
  • Improving SAE dimensionality to better align features with human intuition.

Conclusion: The paper demonstrates that LLM refusal is not a monolithic behavior but a distributed, redundant, and non-linearly interacting set of features. Understanding and manipulating this latent space offers a path toward more transparent, auditable, and robust AI safety mechanisms.