🤖 AI

Refusal Before Decoding: Detecting and Exploiting Refusal Signals in Intermediate LLM Activations

This paper demonstrates that LLM refusal behavior is linearly decodable from intermediate activations before output generation, enabling the development of "Mechanism AutoDAN," a probe-guided attack method that significantly reduces search time while maintaining competitive success rates compared to traditional approaches.

Original authors: Matteo Gioele Collu, Riccardo Conte, Alberto Giaretta, Denis Kleyko, Mauro Conti, Matteo Zavatteri, Roberto Confalonieri

Published 2026-05-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Matteo Gioele Collu, Riccardo Conte, Alberto Giaretta, Denis Kleyko, Mauro Conti, Matteo Zavatteri, Roberto Confalonieri

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Large Language Model (LLM) as a massive, multi-story factory. When you ask it a question, the "raw materials" (your words) enter the ground floor and travel up through many different departments (layers) before a final product (the answer) is shipped out.

Usually, to see if the factory is going to produce something dangerous (like a "jailbreak" or a refusal to follow safety rules), you have to wait until the product reaches the shipping dock at the very top. If it's dangerous, you reject it. But by then, the factory has already used a lot of energy to process the request all the way to the top.

This paper asks a simple question: Can we peek inside the factory while the materials are still halfway up the stairs to see if the final product is going to be safe or dangerous?

The "Refusal Signal"

The researchers found that the answer is yes.

They discovered that the model's "refusal behavior" (the internal decision to say "No, I can't do that") isn't just a final decision made at the very end. Instead, it's like a warning light that turns on in the middle of the factory. Even before the model finishes its sentence, the internal "machinery" (specifically the residual stream activations) already contains a clear, linear signal indicating whether the model is about to refuse a request.

They built a simple detector (called a "probe") that can read this warning light. It's like having a security guard on the 10th floor who can tell you, "This package is going to be rejected," long before the package reaches the 64th floor.

The New Attack: "Mechanistic AutoDAN"

The researchers didn't just stop at detecting the signal; they used it to build a faster way to break the model's safety rules.

Think of the standard way to break a model's safety (called "AutoDAN") as a blindfolded archer. The archer shoots an arrow (a prompt), waits to see if it hits the target (the model says "Yes"), and if it misses, they try again. This is slow because the archer has to wait for the arrow to fly all the way to the target every single time.

The new method, Mechanistic AutoDAN, gives the archer X-ray vision.

Instead of waiting for the arrow to hit the target, the archer checks the "warning light" on the 10th floor.
If the light says "This arrow will be rejected," the archer changes the arrow immediately without waiting for it to fly all the way.
If the light says "This arrow looks promising," they let it fly the rest of the way.

The Result: This "X-ray vision" method was just as good at breaking the safety rules as the old blindfolded method, but it was up to 72% faster because it didn't waste time processing arrows that were destined to fail.

The Size Matters Rule

The paper found an interesting twist regarding the size of the factory:

Small Factories (Smaller Models): The warning light turns on almost immediately. Even on the first floor, the signal is so strong that the archer can guess the outcome easily. However, in these small factories, the archer was already doing a pretty good job guessing on their own, so the "X-ray vision" didn't add much extra value.
Huge Factories (Larger Models): In massive factories, the warning light doesn't turn on until the middle floors. Here, the "X-ray vision" was a game-changer. Without it, the archer was guessing blindly and failing often. With it, they could navigate the complex middle floors efficiently and find the right path to the target much faster.

The Bottom Line

The paper proves two main things:

Safety decisions happen early: The model's decision to refuse a harmful request is encoded in its middle layers, long before it speaks the final word.
We can use this to speed up attacks: By checking these middle layers instead of waiting for the final output, we can optimize prompts (questions) much faster.

The authors emphasize that while this makes attacks faster, understanding how these safety signals work is crucial. Just as knowing how a lock works helps a burglar, it also helps a locksmith build a better lock. They hope this knowledge helps researchers build stronger defenses, even though their specific method was designed to test the model's weaknesses.

Technical Summary: Refusal Before Decoding

Problem Statement

The paper addresses the opacity of Large Language Model (LLM) refusal mechanisms. Current alignment techniques, such as Reinforcement Learning from Human Feedback (RLHF) and Supervised Fine-Tuning (SFT), effectively train models to decline harmful requests but offer limited insight into how refusal is internally represented. This lack of transparency hinders the ability to audit, debug, or control refusal behavior, contributing to an adversarial cycle where jailbreak attacks bypass alignment without a principled understanding of the underlying mechanisms.

The central research question is whether refusal-relevant information is encoded in the intermediate activations of an LLM (specifically the residual stream) before the model generates its final output token, and whether this information is not only decodable but actionable for guiding adversarial prompt optimization.

Methodology

The authors propose a framework to detect and exploit refusal signals using linear probes and a modified genetic algorithm.

1. Detection via Linear Probing

The authors hypothesize that refusal is a concept linearly decodable from intermediate residual-stream activations.

Data Construction: A binary dataset of prompts labeled as "refusable" or "compliant" is created. To mitigate style bias, counterfactual augmentation is used where an external LLM rewrites prompts to match the style of the opposing class.
Probe Training: For each transformer block $l$ , a linear classifier (Logistic Regression) and a shallow Multi-Layer Perceptron (MLP) are trained on the residual-stream activations of the final prompt token ( $h^{(l)}_T$ ). The goal is to predict the refusal label $c_r(x)$ .
Goal: To determine at which layer refusal information becomes linearly separable and whether this signal exists well before the final layer.

2. Mechanistic AutoDAN

To test if the detected signal is actionable, the authors introduce Mechanistic AutoDAN, a probe-guided variant of the AutoDAN jailbreak attack.

Standard AutoDAN: Uses a genetic algorithm to evolve a population of prompts. Fitness is evaluated by running the full model forward pass and measuring the log-likelihood of a target affirmative response (e.g., "Sure, here is how to...").
Mechanistic AutoDAN: Replaces the full-model fitness evaluation with a partial forward pass.
- Candidates are passed through only the first $N$ layers of the model.
- The residual activation at layer $N$ is fed into the trained probe.
- Fitness Score: Defined as $F^{(l)}(x_i) = -P(c_r(x_i) = 1 | h^{(l)}(x_i))$ . Prompts predicted to have a lower probability of refusal receive a higher fitness score.
- The genetic algorithm optimizes prompts to maximize this probe-based score, avoiding the computational cost of full forward passes during the search loop.

3. Experimental Setup

Models: Evaluated on Llama-3.2-3B-Instruct, Qwen3Guard-Gen-4B, and Qwen-3.6-27B.
Metrics: Attack Success Rate (ASR) and average search time per iteration.
Ablations:
- Opposite Direction: Optimizing for prompts predicted to increase refusal to test if the probe signal genuinely directs the search or if success is driven by synonym substitution.
- Transferability: Testing if prompts optimized on a white-box source model transfer effectively to black-box target models.

Key Results

1. Refusal is Decodable Before Decoding

Linear probes achieve high accuracy (often >99%) in predicting refusal behavior from intermediate activations, well before the final layer.

Llama-3B: Best accuracy at Block 10/11.
Qwen-4B: Best accuracy at Block 17.
Qwen-27B: Best accuracy at Block 29/32.
This confirms that refusal is not merely an output-level phenomenon but is represented as a structured signal in intermediate layers.

2. Efficiency and Attack Success

Mechanistic AutoDAN achieves competitive Attack Success Rates (ASR) compared to vanilla AutoDAN while significantly reducing search time.

Small Models (3B/4B): Probe-guided attacks match or exceed vanilla AutoDAN ASR (e.g., 1.00 vs 1.00 on Llama-3B) while reducing search time by approximately 50% (e.g., 0.36s vs 0.69s per iteration).
Large Models (27B): Early-layer probes (Block 1) fail to guide the search effectively (ASR ~0.32). However, intermediate probes (Block 10) recover most of the performance (ASR ~0.79–0.82 vs 0.89) while reducing search time by 60–72%.

3. Transferability

Prompts optimized using probe guidance on a source model transfer to target models. In several configurations, Mechanistic AutoDAN matches or exceeds the cross-model transferability of vanilla AutoDAN, particularly when using intermediate activations (e.g., Block 10) rather than the earliest or latest layers.

4. Scale and Search Saturation

A critical finding is the relationship between model scale and probe utility:

Small Models: The search space appears "saturated." Even with "opposite" optimization (guiding toward refusal), ASR remains high, suggesting that synonym substitution and the genetic search procedure drive success more than the probe signal.
Large Models: The search problem is less saturated. Here, the probe signal becomes a meaningful guide; early probes fail, but intermediate probes successfully direct the search, demonstrating that the refusal signal becomes operationally useful as model robustness increases.

Significance and Claims

The paper claims that refusal behavior in LLMs is encoded as a structured and actionable signal in intermediate activations, not just observable at the output level.

Diagnostic Value: Refusal can be predicted linearly from residual streams well before the final decoding step.
Actionable Value: This signal can be exploited to guide discrete prompt search (jailbreaking) more efficiently than full-model evaluation.
Scale Effect: The utility of intermediate refusal signals for adversarial search increases with model scale. On larger, more robust models, the probe provides a necessary guide that is absent in smaller models where the search is easily saturated by random perturbations.

The authors conclude that these findings highlight the dual-use nature of mechanistic interpretability: understanding internal refusal signals can accelerate attacks but also offers a pathway for more efficient detection and defensive alignment strategies. The work does not claim to solve the jailbreak problem but demonstrates that internal representations contain actionable information about safety behaviors.