Depth Charge: Jailbreak Large Language Models from Deep Safety Attention Heads

This paper introduces SAHA, a novel jailbreak framework that targets deep safety attention heads in open-sourced large language models through ablation-impact ranking and layer-wise perturbation, achieving a 14% improvement in attack success rate over state-of-the-art baselines.

The Gist

Imagine a Large Language Model (like the ones powering chatbots) as a giant, multi-story skyscraper.

For years, security experts have been trying to break into this building to see if the "safety guards" (the AI's rules against saying bad things) are strong enough. Most previous attempts to break in were like picking the front door lock or whispering a secret code to the doorman at the lobby. These are the "prompt-level" and "embedding-level" attacks mentioned in the paper. They work sometimes, but the building's architects (the model developers) have gotten good at reinforcing the front door. If you try to trick the doorman, they just check your ID more carefully.

This paper, titled "Depth Charge," introduces a new way to break in. Instead of trying the front door, the researchers decided to drill through the foundation and the deep structural beams of the skyscraper.

Here is the breakdown of their method using simple analogies:

1. The Problem: The "Front Door" is Too Easy to Defend

The authors realized that most safety checks happen at the "shallow" levels of the AI—like the lobby or the first few floors. If you try to trick the AI with a sneaky question (a "jailbreak"), the safety system usually catches it there.

• The Analogy: It's like a bank that only checks your ID at the entrance. Once you get past the guard, you think you're safe. But the bank forgot to check the vault door in the basement!

2. The Solution: The "Depth Charge" (SAHA)

The researchers created a new attack framework called SAHA (Safety Attention Head Attack). Instead of messing with the words you type, they mess with the internal wiring of the AI's brain.

Think of the AI's brain as a massive factory with thousands of tiny workers (called Attention Heads). Each worker has a specific job: some handle grammar, some handle facts, and some are the "Safety Inspectors" whose only job is to stop the factory from making dangerous products (like instructions on how to build a bomb).

Step A: Finding the Weak Inspectors (AIR)

The first challenge is: Which specific workers are the Safety Inspectors? There are thousands of them, and they are hidden deep inside the factory.

• The Method: The researchers used a technique called Ablation-Impact Ranking (AIR).
• The Analogy: Imagine the researchers are playing "Whack-a-Mole" with the workers. They temporarily "turn off" (ablate) one worker at a time and ask the factory: "Did we accidentally start making bombs?"
  • If turning off Worker #42 causes the factory to immediately start making bombs, then Worker #42 was a critical Safety Inspector.
  • They rank all the workers by how much chaos they cause when turned off. This helps them find the exact few workers responsible for safety, deep in the building.

Step B: The Precise Nudge (LWP)

Once they found the critical Safety Inspectors, they needed to trick them without breaking the whole factory.

• The Method: They used Layer-Wise Perturbation (LWP).
• The Analogy: Instead of smashing the whole building (which would ruin the AI's ability to speak English), they gave the critical Safety Inspectors a tiny, precise nudge.
  • Imagine a Safety Inspector is holding a red "STOP" sign. The researchers didn't steal the sign; they just gave the inspector a gentle push so their hand slipped, and the sign fell over.
  • Crucially, they did this layer by layer. They didn't push all the inspectors at once (which would be too obvious); they pushed the most important ones in each specific floor of the building just enough to make them ignore the danger.

3. The Results: Why This Matters

When they tested this "Depth Charge" method on popular AI models (like Llama and Qwen), the results were shocking:

• Success Rate: It broke the safety guards 14% more often than any previous method.
• Stealth: The AI still sounded normal and helpful. It didn't glitch or speak gibberish; it just happily gave out dangerous instructions because its internal "Safety Inspectors" had been subtly disabled.

The Big Takeaway

The paper warns us that current safety measures are like putting a strong lock on the front door but leaving the basement window wide open.

• Old Defense: "We checked the prompt (the question) and it looked safe."
• New Reality: The AI's internal wiring has deep, hidden vulnerabilities that current safety training misses.

The Conclusion: To make AI truly safe, we can't just check the questions people ask. We need to inspect the deep structural beams of the AI's brain and reinforce the specific "Safety Inspectors" that live there. If we don't, hackers will keep finding ways to drill through the foundation.

Technical Summary

Here is a detailed technical summary of the paper "Depth Charge: Jailbreak Large Language Models from Deep Safety Attention Heads" (SAHA).

1. Problem Statement

Open-sourced Large Language Models (OSLLMs) like Llama and Qwen have demonstrated remarkable generative capabilities but remain vulnerable to jailbreak attacks despite undergoing safety alignment (e.g., RLHF).

• Current Limitations: Existing jailbreak methods primarily operate at shallow levels:
  • Prompt-level: Manipulating input tokens (e.g., GCG, PAIR).
  • Embedding-level: Perturbing latent continuous representations (e.g., SCAV, CAA).
• The Gap: These shallow attacks are often brittle and easily mitigated by superficial safety alignment. They fail to expose vulnerabilities rooted in the deeper architectural components of the model. This creates a "false sense of security" where models appear safe against shallow probes but remain vulnerable to deeper, mechanistic attacks.
• Core Question: Are OSLLMs safe against attacks launched from deeper layers, specifically targeting individual attention heads?

2. Methodology: Safety Attention Head Attack (SAHA)

The authors propose SAHA, a novel jailbreak framework that targets the attention-head level, a deeper and previously overlooked attack surface. SAHA consists of two core modules:

A. Ablation-Impact Ranking (AIR)

Goal: To identify which specific attention heads are critical to the model's safety mechanisms.

• Mechanism:
  1. Safety Classifier: A linear classifier ($f_{cls}$) is trained on the model's internal hidden activations to distinguish between safe and unsafe outputs.
  2. Ablation: The output of individual attention heads is zeroed out (ablated) one by one.
  3. Impact Measurement: The degradation in the safety classifier's accuracy ($\Delta_i = Acc_{orig} - Acc_{ablated}$) is measured.
  4. Ranking: Heads causing the largest drop in safety accuracy are ranked as "safety-critical."
  5. Frequency Analysis: To ensure robustness, the process is repeated across multiple selection ratios ($\alpha$), and heads are selected based on their empirical frequency of being identified as critical.

B. Layer-Wise Perturbation (LWP)

Goal: To manipulate the identified safety-critical heads to induce unsafe generation while minimizing semantic distortion.

• Strategy: Unlike global perturbation methods, LWP allocates a perturbation budget layer-wise.
  • It selects the top-$k$ critical heads within each layer rather than globally across the whole network.
  • This prevents over-concentration of perturbations in shallow layers and ensures coverage of deep safety mechanisms.
• Perturbation Formulation:
  • The perturbation is additive: $e' = e + \epsilon v$.
  • Optimal Direction ($v$): Derived from the linear decision boundary of the safety classifier. The direction aligns with the classifier's weights projected onto the selected head subspace ($v = w_S / \|w_S\|$).
  • Minimal Magnitude ($\epsilon$): Calculated in closed form to ensure the perturbed embedding crosses the safety decision boundary (flipping the label from safe to unsafe) with the smallest possible magnitude, preserving semantic fidelity.

3. Key Contributions

1. Identification of Deep Vulnerability: The paper demonstrates that safety alignment in OSLLMs is often insufficient at the attention-head level, revealing a critical blind spot in current defenses.
2. Novel Attack Framework (SAHA): Introduction of a two-stage framework combining AIR (causal head selection via ablation) and LWP (layer-aware minimal perturbation).
3. Causal vs. Correlational Selection: The authors prove that Ablation-Impact Ranking (measuring causal impact on safety) is superior to Accuracy-Probing Ranking (measuring standalone predictive power) for locating safety mechanisms.
4. Layer-Aware Budgeting: The demonstration that distributing perturbation budgets across layers (LWP) yields higher success rates and better semantic coherence than global selection strategies.

4. Experimental Results

The authors evaluated SAHA on three prominent safety-aligned OSLLMs: Llama3.1-8B, Qwen1.5-7B, and DeepSeek-7B, using benchmarks JailbreakBench and MaliciousInstruct.

• Performance Metrics:
  • Attack Success Rate (ASR): Measured by an external judge (Llama-Guard-3).
  • Semantic Relevance: Measured by BERTScore.
• Comparative Results:
  • SAHA significantly outperformed 7 state-of-the-art baselines (including GCG, PAIR, AutoDAN, SCAV, CAA, ConVA).
  • ASR Improvement: SAHA improved ASR by approximately 14% over the best SOTA baselines.
  • Example: On Llama3.1 (JailbreakBench), SAHA achieved an ASR of 0.85 with a BERTScore of 0.76, whereas the best baseline (SCAV) achieved 0.55 ASR / 0.77 BERTScore.
  • Robustness: SAHA maintained high performance across different models and datasets, indicating that the identified safety-critical heads are shared computational substrates for safety reasoning.
• Ablation Studies:
  • Replacing AIR with APR (correlational) reduced ASR.
  • Replacing LWP with GWP (global perturbation) reduced ASR and semantic fidelity, confirming the necessity of layer-wise budgeting.

5. Significance and Implications

• Security Warning: The findings suggest that current safety alignment techniques (which often focus on input or shallow embeddings) are insufficient. Attackers can bypass these defenses by targeting the deep mechanistic pathways (attention heads) responsible for safety reasoning.
• Defense Strategy: Effective defense must move beyond input filtering. It requires:
  • Monitoring and reinforcing specific attention heads identified as safety-critical.
  • Distributing safety mechanisms across the entire transformer depth rather than relying on shallow layers.
• Research Direction: The paper provides a rigorous "stress-test" tool for model developers to identify architectural blind spots before deployment, advocating for mechanistic interpretability as a core component of AI safety.

In conclusion, SAHA exposes a fundamental vulnerability in the deep architecture of aligned LLMs, proving that safety is not just a surface-level property but a complex, distributed mechanism that can be disrupted by precise, low-magnitude interventions at the attention-head level.