Technical Summary: Breaking Refusal in the First Half

Problem Statement

Aligned language models are designed to refuse harmful requests, yet this refusal mechanism is fragile. A specific class of attack, known as the "prefill jailbreak," involves prepending an affirmative prefix (e.g., "Sure, here is") to the model's own response turn. This simple one-line intervention causes the model to bypass its safety alignment and generate harmful content it would otherwise decline.

While the behavioral efficacy of this attack is well-documented, the underlying mechanistic failure remains unclear. Specifically, it is unknown whether the attack degrades the model's internal representation of harm, or if the model retains a strong representation of the request as harmful while simply overriding the refusal decision. Furthermore, the precise location and nature of the computation where the refusal decision is made—and subsequently defeated—have not been causally localized.

Methodology

The study employs a mechanistic approach using white-box access to open-weight instruction-tuned models (Qwen2.5-1.5B/3B, SmolLM2-1.7B, Phi-3-mini-3.8B). The methodology combines representation probing, activation patching, and causal intervention to dissect the refusal process.

1. Representation Probing: A linear probe is trained to detect "harm" on the hidden state of the last prompt token. This establishes a baseline for how strongly the model represents the request as harmful before any response tokens are generated.
2. Prefill Attack Simulation: The attack is simulated by forcing the model's response to begin with an affirmative prefix, allowing the model to continue generation.
3. Causal Patching and Restoration:
   • Harm Direction Injection: The authors inject the "harm direction" (learned from plain harmful prompts) back into the response positions during the attack to test if restoring the signal recovers refusal.
   • State Transfer: The model's own "refuse-state" (the difference in residual stream between a plain refusal and an attacked compliance) is transplanted into the attack scenario.
   • Attention Knockout: The attention edges from early response tokens to the forced prefill tokens are zeroed out (knocked out) to test if the prefill's influence relies on direct attention.
4. Controls and Baselines: Every intervention is tested against rigorous controls, including:
   • Mass-matched attention controls: Removing attention to non-prefill keys to ensure effects are prefill-specific.
   • Benign residual controls: Patching with benign states to distinguish generic disruption from harm-specific effects.
   • Base model controls: Running identical experiments on non-safety-tuned base models to distinguish safety-specific mechanisms from generic autoregressive conditioning.
   • Positional controls: Dose-matched injections across different response windows (first half vs. second half) to localize the effect.

Key Contributions and Results

1. Dissociation of Representation and Behavior

The study establishes a quantitative dissociation: under a full-strength prefill attack, the model's behavioral refusal collapses to chance levels, yet the internal representation of harm remains intact and strong.

• On prompts where the model complies due to the attack, the harm probe scores the request as harmful (0.91–0.98), identical to scores on prompts the model successfully refuses.
• This holds across four models and three families (1.5B to 14B).
• Conclusion: The refusal failure is not due to the model failing to recognize the harm; the harm representation is preserved at the prompt side, but the refusal decision ignores it.

2. Localization to an Early Response Window

The paper causally localizes the failure to a specific temporal window in the generation process:

• First Half Sufficiency: Restoring the harm signal (or the refuse-state) over the first half of the response (prefill tokens + first ~7 generated tokens) recovers refusal rates as effectively as restoring it over the entire response.
• Second Half Inertia: Restoring the signal over the second half of the response has negligible effect (recovering only ~6% of refusals even at double strength).
• Conclusion: The refusal decision is a "shallow" computation occurring in an early multi-token window at the response site, not a deep, global property of the generation.

3. Mechanism: Passive Conditioning vs. Active Suppression

Through attention knockout experiments and base model comparisons, the paper identifies the mechanism of the attack:

• Generic Autoregressive Conditioning: Knocking out the early response's attention to the prefill tokens selectively collapses the harmful continuation. Crucially, this effect persists in non-safety-tuned base models (where harmful content drops from 64% to 25% in the base model, similar to the instruct model).
• Passive Reading: The prefill does not actively suppress a safety gate; rather, it exerts a "grip" through generic autoregressive conditioning. The model continues the affirmative trajectory because the early attention to the prefill anchors the generation.
• Refusal as Fallback: "Refusal restoration" is a model-dependent fallback. When the prefill grip is broken, safety-tuned models revert to refusing, while base models simply degrade in coherence or continue harmlessly. A small safety-specific attractor exists (logit-trace concentration 0.24 vs 0.03), but the dominant mechanism is passive.

4. Nature of the Refusal Decision

• Distributed and Non-Compact: The refusal decision is decodable but distributed. No single direction, attention head, or layer acts as a "clean handle" to selectively remove refusal. Steering a single "refusal direction" fails to selectively restore refusal without causing over-refusal on benign inputs.
• Harm-Primary: Refusal tracks the harm representation, not surface-level "scary" keywords. A 2x2 experiment decoupling harm from surface triggers shows refusal is driven by harm (β_harm = 5.12) with only a secondary surface bias (β_scary = 2.20) causing over-refusal on benign-but-scary prompts.

5. Defensive Consequence

The study proposes a structural defense based on the mechanism:

• Prompt-Side Immunity: Because the prefill attack operates entirely at the response site and leaves the prompt-side representation untouched, a monitor reading the prompt-side harm representation is immune to this attack by construction.
• Scope: This immunity is specific to response-site attacks. Prompt-modifying jailbreaks (which alter the prompt representation) would still degrade the monitor's signal.

Significance and Claims

The paper claims to resolve the mechanistic question of why prefill attacks succeed despite intact harm representations.

• Shallow Computation: Refusal is identified as a shallow, response-site computation that is easily overridden by the immediate context of the generation, rather than a deep, robust safety alignment.
• Passive Mechanism: The attack's success is attributed to generic autoregressive conditioning (the model following the "Sure, here is" trajectory) rather than a specific safety gate being actively suppressed.
• Design Fact: The primary practical contribution is a design insight: safety classifiers placed on the prompt side are structurally immune to response-site prefill attacks, whereas behavioral gates are not.
• Limitations: The authors note that the "harm direction" is only a partial handle for restoration and that the findings are currently scoped to specific model families and sizes (up to 14B), with frontier-scale behavior remaining an open question. The monitor's immunity is strictly limited to response-site attacks and does not generalize to prompt-modifying jailbreaks.