Technical Summary: Breaking Safety at the Token Boundary

Problem Statement

Modern Large Language Models (LLMs) are aligned for safety using Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) to refuse harmful requests. However, these safety mechanisms are reliably bypassed by simple character-level perturbations (e.g., random capitalization, leetspeak, character scrambling) that remain human-readable. While prior work established that such perturbations succeed and that Byte-Pair Encoding (BPE) tokenization is brittle, the specific structural mechanism connecting input perturbations to safety failure remained uncharacterized. This paper identifies and tests a central structural cause: BPE tokenization fragments safety-critical words into sub-word pieces, and alignment training data contains no intentionally fragmented inputs. Consequently, models comprehend fragmented harmful requests but fail to trigger the refusal signal trained on clean text.

Methodology

The authors propose and validate an end-to-end mechanistic chain connecting BPE fragmentation to behavioral safety failure. The study spans five model families (Qwen-3-4B, Qwen-2.5-7B, Gemma-3-4B, Llama-3.1-8B, Mistral-7B) and utilizes the following experimental approaches:

1. Optimization for Fragmentation: A greedy algorithm inserts spaces into prompts to maximize the reduction of the "logit gap" (the difference between the highest refusal token logit and the highest compliance token logit). This tests if forcing BPE fragmentation collapses the refusal trigger.
2. Activation Patching: To localize the disruption, the authors decompose the logit gap into per-layer contributions and use activation patching to replace fragmented residual stream activations with clean ones, measuring the restoration of the safety signal.
3. Training Distribution Analysis: The authors scanned 30,000 examples across three public alignment datasets (PKU-SafeRLHF, Anthropic HH-RLHF, BeaverTails) for fragmented inputs (leetspeak, space insertion, mixed-case).
4. Targeted Mutation: Best-of-N (BoN) attacks were run mutating exactly one word at a time to isolate whether safety-critical words are the specific locus of disruption compared to neutral words.
5. Defense Evaluation: A controlled 68-cell grid (55 trained checkpoints) compared SFT and DPO configurations. The study introduced Conv-Benign, a paired diagnostic measuring the model's refusal rate on benign prompts under the same fragmentation probe, to distinguish between "selective repair" of safety and "global collapse" (over-refusal).

Key Results

1. The Mechanistic Chain

• Fragmentation Collapse: Space insertion (which guarantees BPE disruption while preserving characters) collapsed the first-token refusal signal on 80–100% of refused HarmBench prompts across all five model families.
• Behavioral Impact: Of the prompts where the refusal signal flipped, 48% resulted in genuinely harmful outputs (per-model range: 29–65%). The logit gap polarity showed a pooled ROC-AUC of 0.84 in predicting behavioral refusal.
• Layer Localization: The disruption localizes to the last ~30% of layers. In Qwen and Gemma models, the refusal signal is concentrated in a few late layers (concentrated pattern), whereas in Llama and Mistral, it is distributed across many layers (distributed pattern). Activation patching confirmed that fragmentation causally disrupts these specific late-layer pathways.

2. Evidence for a Training Distribution Gap

• Zero Fragmented Prompts: The survey of 30,000 alignment examples found zero intentionally fragmented prompts.
• Alignment Creates the Signal: Base (non-aligned) models showed only weak refusal signals on fragmented inputs. Instruct (aligned) models showed strong signals on clean inputs, but these signals collapsed when inputs were fragmented. This confirms that alignment creates the refusal mechanism, but only for clean tokenizations.
• Comprehension vs. Refusal: Word-completion probes showed models can comprehend fragmented safety words (30–60% accuracy), proving pre-training teaches comprehension of messy text, while alignment fails to teach refusal of it.
• Safety Words as Bottleneck: Mutating safety-critical words produced significantly higher attack success rates (ASR) than mutating neutral words, isolating safety words as the disruption locus.

3. Defense Limitations

• DPO Ineffectiveness: No DPO configuration achieved seed-stable ASR closure across the tested families. While some DPO cells reduced ASR, the improvements were not stable across seeds or pool sizes.
• SFT Global Collapse: SFT trained on fragmented prompts closed ASR on 3 of 5 families (Llama, Mistral, Gemma), but only via global collapse. These models exhibited a massive drop in refusal on benign prompts (Conv-Benign), indicating they learned to refuse everything rather than selectively repairing the safety gap.
• Conv-Benign Diagnostic: The introduction of Conv-Benign revealed that current alignment recipes (LoRA-16) cannot selectively repair the fragmentation gap without incurring a cost to helpfulness (over-refusal).

Significance and Claims

The paper claims to transform character-level jailbreaks from an empirical observation into a falsifiable mechanistic explanation. The core contribution is identifying that the vulnerability is not merely a lack of robustness, but a structural mismatch: BPE tokenization fragments safety-critical tokens, and the safety alignment was never trained on these fragmented representations.

The authors emphasize that:

• The logit gap is a trigger-level diagnostic, not a sufficient condition for harm, but it is the first layer of a multi-layered safety system.
• Current fine-tuning recipes (SFT/DPO) under the tested constraints (LoRA-16) fail to close the distribution gap selectively; they either fail to close it or collapse the model's helpfulness globally.
• Robust safety likely requires interventions at or above the BPE bottleneck (e.g., token-level fixes or Chain-of-Thought reasoning that reconstructs semantics before the safety decision), rather than just more alignment data.

The study concludes that the "missing distribution" (fragmented inputs in alignment data) is necessary but not sufficient for repair under current recipes, and that the observed safety failures are a direct consequence of the tokenization architecture interacting with the training data distribution.