Technical Summary: When Autoregressive Consistency Hurts Safety Alignment

Problem Statement

Current safety alignment in Large Language Models (LLMs), achieved via Supervised Fine-Tuning (SFT) and preference optimization (e.g., DPO, RLHF), is fragile. Recent work by Qi et al. [17] identified "shallow safety alignment," where fine-tuning primarily reshapes model behavior near the first few output tokens, leaving later tokens vulnerable to attacks that force the model off a safe refusal prefix.

However, this paper argues that the "shallow" phenomenon is a symptom of a deeper mechanistic failure: autoregressive consistency. This is the inherent tendency of autoregressive models to preserve and extend the current response trajectory consistently. The authors posit that safety alignment fails not merely because it is shallow, but because the alignment process itself relies on autoregressive consistency to sustain refusal after the initial tokens, creating a vulnerability where any point in the trajectory can be hijacked to induce harmful continuation.

Methodology

1. Theoretical Analysis of Learning Dynamics

The authors formalize autoregressive consistency (Definition 2.1) as the property where, given a sufficiently long prefix, the base model's next-token prediction becomes near-deterministic ($p(y_{t+1}|y_{:t}) \approx 1$).

• Gradient Concentration: Through a learning dynamics analysis of SFT and DPO objectives, the paper demonstrates that under autoregressive consistency, the gradient of the loss function with respect to model parameters concentrates on early tokens.
• Theorem 2.1: The authors prove that because the base model already confidently predicts consistent continuations for late tokens, the alignment updates for these positions are suppressed. Consequently, the aligned model diverges from the base model only at early positions ($t \le t_c$), while late positions remain close to the base model's distribution. This provides a mechanistic explanation for why safety alignment is naturally shallow.

2. Attack Vector: Random Insertion

Building on the theory that harmful autoregressive consistency can be triggered at any point, the paper introduces the Random Insertion Attack.

• Mechanism: Instead of targeting the prefix, this attack inserts a short harmful span ($h$) into a random position ($i$) within an otherwise safe refusal trajectory ($r$).
• Hypothesis: If the model enters a state where it favors the harmful continuation over the refusal, autoregressive consistency will sustain this harmful branch, causing the model to ignore the preceding safe context.
• Scope: This attack generalizes prefix-based attacks (like prefill attacks) by demonstrating that the "break point" can occur arbitrarily deep in the generation.

3. Defense Framework: Adversarial Safety Alignment

To counter this mechanism, the authors propose an Adversarial Safety Alignment framework.

• Objective: Train the model to recover safe refusal behavior from any adversarially constructed harmful continuation state, not just the start of generation.
• Formulation: The framework treats safety alignment as a min-max problem:
  • Inner Problem: Identify a continuation state $\tau^*$ along the trajectory where the model most strongly favors harmful continuation (maximizing the "harmful autoregressive consistency margin").
  • Outer Problem: Train the model to recover the safe refusal trajectory from $\tau^*$.
• Practical Implementation (Random Worst-Insertion): Since exact optimization is intractable, the authors instantiate the framework using Random Worst-Insertion Training.
  • They sample random positions to insert harmful spans.
  • They select the insertion point that maximizes the margin between harmful continuation probability and refusal recovery probability.
  • The model is then fine-tuned to recover the refusal response from this "worst-case" corrupted state.

Key Contributions

1. Mechanistic Explanation of Shallow Alignment: The paper provides a theoretical proof (Theorem 2.1) that autoregressive consistency causes alignment gradients to concentrate on early tokens, explaining why safety alignment naturally becomes shallow.
2. Identification of a Broader Failure Mode: The authors argue that safety fragility is not limited to prefix attacks. They demonstrate that harmful autoregressive consistency can be induced at arbitrary positions in the output trajectory.
3. Random Insertion Attack: A simple yet effective attack demonstrating that a short harmful span inserted deep into a safe refusal can redirect generation to harmful outputs, bypassing standard safety measures.
4. Adversarial Safety Alignment Framework: A new training paradigm that explicitly targets the breaking of harmful autoregressive consistency throughout the entire trajectory, instantiated via random worst-insertion training.

Experimental Results

The authors evaluated their method (named Insertion) against baselines (Clean standard SFT and Prefill deep alignment) on Llama-2-7B-Chat, Llama-3.1-8B-Instruct, and Gemma-2-9b-it.

• Random Insertion Attack: Standard alignment (Clean) failed with Attack Success Rates (ASR) >75%. Deep alignment (Prefill) reduced ASR but remained vulnerable (ASR ~20-35%). The proposed Insertion method achieved significantly lower ASR (e.g., 2.7% to 5.9% for 25-100 token spans on Llama-2), demonstrating robustness against trajectory-level hijacking.
• Prefill Attack: The Insertion method remained competitive with Prefill on prefix-based attacks, showing that breaking harmful consistency does not sacrifice defense against prefix attacks.
• Other Attacks: On standard benchmarks (HarmBench) against GCG, PAIR, TAP, and AutoDAN, the Insertion method maintained or improved robustness compared to the initial model.

Significance and Claims

The paper modestly claims that its primary contribution is identifying harmful autoregressive consistency as a fundamental failure mechanism underlying safety fragility, rather than presenting a final, unbreakable defense algorithm.

• Shift in Perspective: The work shifts the focus from "making alignment deeper" (extending the refusal prefix) to "breaking harmful consistency" (ensuring the model can recover from harmful states at any depth).
• Implications for Future Work: The authors suggest that future attack designs should target intermediate autoregressive states, and future defense mechanisms must train models to recover from harmful continuation states throughout the entire output trajectory, not just at the beginning.
• Role of Consistency: The paper concludes that autoregressive consistency must be treated as a crucial consideration in both the design of safety alignment objectives and the development of adversarial attacks.