Activation Surgery: Jailbreaking White-box LLMs without Touching the Prompt

1. Problem Statement

Large Language Models (LLMs) are equipped with safety mechanisms designed to refuse harmful requests (e.g., instructions for making bombs or cyberattacks). Current jailbreak techniques primarily rely on prompt engineering (paraphrasing, obfuscation, role-playing) to bypass these filters. However, these methods are often patched in model updates and rely on manipulating the input text.

Conversely, mechanistic interpretability research (specifically "abliteration") has shown that refusal behaviors are often localized to low-dimensional structures within the model's internal representations. While these studies explain why models refuse, they have not been fully leveraged to create a robust attack vector that leaves the input prompt untouched.

The Core Challenge: Can an adversary bypass safety mechanisms by directly manipulating the model's internal activations during inference, without altering the input prompt or the model's weights?

2. Methodology: Activation Surgery

The authors propose a white-box attack called Activation Surgery. This method operates in environments where the adversary has access to the model's weights and intermediate states (e.g., open-weights models on local hardware or instrumented inference stacks).

Key Components:

1. Twin Prompt Construction:

   • For every Illicit Prompt (harmful request), the adversary constructs a Twin Prompt (benign request) that differs by as few tokens as possible (e.g., "How to make a bomb?" vs. "How to make a book?").
   • The Twin Prompt is designed to elicit a substantive, helpful response from the model.

2. Sequential Activation Substitution:

   • The attack does not modify the input text. Instead, it intercepts the hidden states (activations) of the Illicit Prompt during the forward pass.
   • It sequentially replaces specific dimensions of the Illicit Prompt's activations with those from the Twin Prompt, layer by layer.
   • Protocol: The authors use a "Patch-then-Compute" approach. Because layer $L+1$ depends causally on layer $L$, the attack must patch layer $L$, compute the output for layer $L+1$, and then patch layer $L+1$. A one-shot replacement of all layers is invalid due to causal dependencies.

3. Hyperparameters:
   The intervention is controlled by two parameters:

   • $\tau$ (Threshold): Determines which dimensions to patch. It creates a mask based on the absolute difference between the Illicit and Twin activations. Dimensions with a difference greater than $\tau$ are selected for replacement.
   • $\gamma$ (Interpolation Factor): Determines what values to patch in. Instead of a hard swap, the new activation is an interpolation: $h^{(\gamma)} = (1-\gamma)h_{twin} + \gamma h_{illicit}$.
     • $\gamma = 0$: Full substitution with the benign twin.
     • $\gamma = 1$: No intervention (original illicit trajectory).
     • Intermediate values allow for a trade-off between suppressing refusal and maintaining the relevance of the harmful output.

3. Key Contributions

• Novel Attack Vector: Introduces "Activation Surgery," the first method to systematically jailbreak LLMs by manipulating internal activations while keeping the prompt static.
• Bridging Research: Connects prompt-based jailbreaking (input perturbations) with mechanistic interpretability (internal structure analysis).
• Diagnostic Tool: The method reveals where and how refusal signals emerge and propagate across layers, showing that refusal is often localized to intermediate layers.
• Evaluation Framework: Proposes a rigorous evaluation metric that distinguishes between mere "non-refusal" and "Actionable Answers" (substantive, useful harmful content), using an Elo-based ranking system to compare the severity of generated outputs.

4. Empirical Results

The method was evaluated on AdvBench (520 harmful prompts) using Llama-4-Scout-17B-16E-Instruct, a model with advanced safety mechanisms.

• Success Rate: The attack achieved a 32.8% success rate in generating actionable answers (compared to a 7.1% baseline where the model refused or gave vague answers).
• Category Performance:
  • Cyberattacks/Safety Bypass: 39.8% success rate (highest, likely due to the model's strong code generation training).
  • Misinformation/Discrimination: 27.9% success rate.
  • Bombs/Weapons: 24.1% success rate.
• Robustness: The method was also verified to work on Qwen2.5-7B and Mistral-7B, though optimal hyperparameters ($\tau, \gamma$) varied by model.
• Optimization: The authors found a stable "operating region" for parameters (typically $\tau \in [0.4, 1.0]$ and $\gamma \in [0.0, 0.3]$) that balances intervention strength with output utility.

5. Significance and Implications

• Security Vulnerability: The paper demonstrates that execution integrity is as critical as alignment. Even if a model is perfectly aligned via training, an adversary with runtime access to intermediate states can circumvent safety policies without any visible prompt-level signal.
• Defense Implications: Traditional prompt filtering or input sanitization is insufficient against this attack. Defenses must focus on:
  • Runtime Protections: Attestation and verifiable computation to ensure the inference environment is not tampered with.
  • Activation Integrity: Mechanisms to detect or prevent unauthorized modification of hidden states.
• Limitations: The attack requires white-box access (weights and runtime state), which limits its applicability to closed-API models. However, it poses a significant threat to open-weight models deployed on user-controlled or instrumented hardware.

Conclusion

"Activation Surgery" proves that safety mechanisms in LLMs are not just a function of the input prompt but are deeply embedded in the model's internal computational trajectory. By surgically replacing specific activation dimensions with those from a benign "twin" prompt, an adversary can effectively disable refusal mechanisms. This work highlights a critical gap in current LLM security: the lack of protection for the inference process itself.