🤖 AI

Robust Harmful Features Under Jailbreak Attacks: Mechanistic Evidence from Attention Head Specialization in Large Language Models

This paper reveals that jailbreak attacks bypass LLM safety not by eliminating all safety features, but by selectively suppressing early-layer Adversarially Compromised Heads while leaving mid-layer Safety-Aligned Heads robustly active, a phenomenon termed "Robust Harmful Features" that enables both mechanistic understanding and effective detection of attacks.

Original authors: Yanchen Yin, Dongqi Han, Linghui Li

Published 2026-06-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yanchen Yin, Dongqi Han, Linghui Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Large Language Model (LLM) as a highly trained security guard at a museum. This guard has been taught to spot dangerous items (harmful requests) and stop them from entering. Usually, if someone asks for something bad, the guard says, "No, I can't do that."

However, "jailbreak" attackers are like clever thieves who try to trick the guard by dressing up their dangerous requests in fancy costumes or speaking in riddles. Sometimes, these tricks work, and the guard lets the bad thing through.

This paper investigates how these tricks work and why the guard sometimes still knows the item is dangerous, even after letting it through.

Here is the breakdown of their findings using simple analogies:

1. The Two Types of "Guards" Inside the Model

The researchers looked inside the model's brain (specifically at its "attention heads," which are like tiny specialized workers). They found that when an attack succeeds, the model doesn't just turn off its safety system completely. Instead, it has two distinct types of workers reacting differently:

The "Confused" Workers (Adversarially Compromised Heads - ACHs):
- Location: These work in the early stages of processing (the front of the line).
- Behavior: When a normal bad request comes in, these workers sound the alarm. But when a "jailbreak" trick comes in, these specific workers get confused or silenced. They stop ringing the bell.
- The Trick: The attacker's costume (the "attack template") specifically targets these workers to shut them up.
The "Stubborn" Workers (Safety-Aligned Heads - SAHs):
- Location: These work in the middle stages of processing.
- Behavior: Even when the attack succeeds and the model generates a bad answer, these workers are still screaming, "This is dangerous!" They keep their alarms ringing loudly, even though the final decision was to let the bad thing through.
- The Discovery: The paper calls this "Robust Harmful Features." It means the model knows it's doing something bad deep down, even if it's pretending to be okay on the surface.

2. How the Attack Works (The "Silencer" Analogy)

Think of the model's safety system as a choir.

Normally, when a bad request comes in, the whole choir sings "NO!"
When a jailbreak attack happens, the attacker doesn't make the choir forget the song. Instead, they use a "silencer" to specifically mute the Confused Workers (the early ones).
Because the early workers are silenced, the final decision gets made without the usual "No" signal.
However, the Stubborn Workers in the middle of the choir are too loud to be silenced. They keep singing "DANGER!" in the background. The attack bypasses the refusal, but it cannot erase the internal knowledge that the request is harmful.

3. Proving the Theory (The "Surgery" Experiment)

To prove this wasn't just a guess, the researchers performed "surgery" on the model:

Silencing the Confused Workers: They manually turned off just a tiny number of the early "Confused" workers (about 8 out of hundreds).
- Result: Suddenly, the model started saying "Yes" to bad requests that it usually refused. This proved that silencing these specific workers is enough to break the safety guard.
Silencing the Stubborn Workers: They turned off the middle "Stubborn" workers.
- Result: The internal "DANGER" alarms stopped ringing as loudly. This proved that these workers were indeed the source of the persistent safety signals.

4. The Practical Result: A "Truth Detector"

Because the "Stubborn Workers" keep screaming "DANGER" even when the model is tricked into saying "Yes," the researchers built a new tool.

How it works: Instead of asking the model "Is this bad?" (which the model might lie about if tricked), this tool just listens to the internal "Stubborn Workers."
The Magic: It can detect that an input is harmful without needing to retrain the model or change its settings. It simply reads the internal signals that the attack failed to hide.
Performance: This tool worked very well, catching harmful content even when the model itself was fooled into generating it.

Summary

The paper reveals that jailbreak attacks are like a "selective silencer." They don't delete the model's safety knowledge; they just temporarily mute the specific parts of the brain that say "No" at the very beginning. The rest of the brain still knows the request is bad. By listening to the parts of the brain that can't be silenced, we can build better detectors that see through the tricks.

Technical Summary: Robust Harmful Features Under Jailbreak Attacks

1. Problem Statement

Jailbreak attacks successfully bypass the safety alignment of Large Language Models (LLMs), inducing them to generate harmful content. However, the internal mechanisms of these attacks remain poorly understood. A fundamental question persists: when a jailbreak succeeds, does the model comprehensively eliminate its safety representations, or does it selectively suppress specific components while leaving others intact? Prior work has focused on identifying safety-relevant neurons, layers, or refusal directions under normal conditions, but the specific interplay between attack mechanisms and internal safety signals during successful bypasses requires further investigation.

2. Methodology

The authors propose a mechanistic interpretability framework to trace information flow and classify attention heads based on their behavior under three input conditions: benign instructions, harmful instructions (which the model refuses), and jailbreak attack instructions (which succeed).

2.1. Back-Projection of Refusal Direction

Building on the finding that refusal behavior is mediated by a single direction in activation space (Arditi et al., 2024), the authors extract a refusal direction $r(L^*)$ at a mid-layer $L^*$ . They then back-propagate this direction through the Output-Value (OV) circuits of attention heads across all layers. Since the OV circuit is a linear transformation, the refusal direction can be precisely projected into the output space of each individual head, quantifying each head's contribution to the refusal signal.

2.2. Activation Collection and Classification

Activation scores are collected for each attention head at the End-of-Instruction (EOI) token across benign, harmful, and attack datasets. The authors employ a two-stage filtering procedure based on Kernel Density Estimation (KDE) and the Overlap Coefficient (OVL):

Harmful vs. Benign: Heads with low overlap are identified as "harmful-salient," meaning they respond differentially to harmful content.
Harmful vs. Attack: Among harmful-salient heads, those with high overlap between harmful and attack inputs maintain stable activation distributions. Those with low overlap exhibit attack-induced shifts.
- Adversarially Compromised Heads (ACHs): Heads where activation is significantly suppressed under attack inputs ( $z_{attack} < z_{harmful}$ ).
- Safety-Aligned Heads (SAHs): Heads where activation is strengthened or remains robust under attack inputs ( $z_{attack} > z_{harmful}$ ).

2.3. Causal Validation via Ablation

To move from correlation to causation, the authors perform intervention experiments:

ACH Ablation: Injecting negative features into identified ACHs on normally refused harmful inputs to simulate attack effects.
SAH Ablation: Suppressing SAHs during attack inputs to measure the impact on mid-layer activation strength.

2.4. Token-Level Attribution

Using the linearity of attention, the authors decompose head activations into contributions from original harmful tokens versus attack-template tokens to understand the source of behavioral differences.

3. Key Findings and Results

3.1. Functional Differentiation of Attention Heads

The study identifies two distinct classes of attention heads that explain the mechanics of jailbreaks:

ACHs (Adversarially Compromised Heads): Concentrated in early layers (e.g., layers 0–3 in Llama-3). These heads are active during normal harmful inputs but are strongly suppressed by attack templates.
SAHs (Safety-Aligned Heads): Distributed in mid-layers (e.g., layers 5–12 in Llama-3). These heads maintain robust, substantial activations even when the model successfully generates harmful content via a jailbreak.

3.2. Causal Mechanisms

ACH Suppression is Sufficient: Ablating a small number of ACHs (e.g., 8 heads in Llama-3-8B) is sufficient to induce jailbreak-like behavior on normally refused inputs, increasing the Attack Success Rate (ASR) from 0% to over 95%. This suggests that attacks bypass refusal by selectively suppressing these critical early-layer components.
SAHs Sustain Robust Features: Ablating SAHs substantially weakens mid-layer activation strength (e.g., a 14–18% drop in mean absolute activation). This confirms SAHs as a primary source of "Robust Harmful Features"—persistent internal representations of harmful semantics that attacks fail to eliminate.

3.3. Token-Level Attribution

Token-level analysis reveals that ACH suppression is driven specifically by negative contributions from attack-template tokens. Conversely, SAHs exhibit strong positive responses to these same attack templates. This explains the phenomenon: attacks do not erase the representation of harmfulness; they selectively suppress the "gatekeepers" (ACHs) while the "sustainers" (SAHs) remain active.

3.4. Practical Validation: Training-Free Detection

Leveraging the persistence of these robust features, the authors propose a training-free detector. By aggregating standardized activations from a broad set of discriminative components (not limited to ACHs or SAHs), the detector achieves competitive performance on safety benchmarks.

Results: On adversarial benchmarks like SALAD-Bench and WildJailbreak, the detector achieves high Macro-F1 scores (e.g., 0.976 on Gemma-2B), significantly outperforming vanilla model refusal and dedicated safety models like LlamaGuard in adversarial settings.
Significance: This demonstrates that internal safety signals remain detectable even when the model's explicit output is compromised.

3.5. Scaling Analysis

Preliminary analysis on Llama-3-70B confirms that the ACH/SAH functional differentiation generalizes to larger models, though the number of required ACH ablations to induce jailbreaks increases with model scale.

4. Significance and Claims

The paper claims to provide a mechanistic account of why jailbreak attacks succeed without completely destroying safety representations. The core contribution is the identification of Robust Harmful Features: internal safety signals that persist under attack, sustained by SAHs, while the bypass is achieved through the selective suppression of ACHs.

The authors emphasize that their findings suggest safety alignment is bypassed rather than broken by attacks. This insight has practical implications for defense:

Mechanistic Understanding: It clarifies that attacks target specific components (ACHs) rather than erasing the entire safety subspace.
Detection: The persistence of robust features enables the development of training-free, white-box detectors that can identify harmful intent even when the model is successfully jailbroken.

5. Limitations

The authors explicitly note several limitations:

Scope: The analysis focuses on attention heads and does not explicitly model MLP layers or LayerNorm, though they argue attention heads are the primary information-routing mechanism.
Refusal Representation: The study uses a single refusal direction as a first-order summary rather than a full refusal subspace.
Data Constraints: The analysis relies on paired samples where attacks succeed against refused requests. For highly aligned models (like Llama-3-70B), successful attack samples are rare (<0.5%), limiting statistical power.
Deployment: The proposed detector requires white-box access to internal activations, limiting direct deployment to open-weight or internal settings.