Technical Summary: Safety Anchor: Defending Harmful Fine-tuning via Geometric Bottlenecks

1. Problem Statement

Large Language Models (LLMs) aligned via Reinforcement Learning from Human Feedback (RLHF) remain vulnerable to Harmful Fine-tuning (HFT). In this threat model, attackers fine-tune a safety-aligned model on a dataset containing malicious instructions (e.g., jailbreak examples) mixed with benign tasks. Even a small proportion of harmful data can strip away safety guardrails, restoring the model's ability to generate harmful content while maintaining performance on benign downstream tasks.

Existing defense mechanisms attempt to constrain the model's internal states during fine-tuning, falling into three categories:

1. Parameter-based: Restricting weight deviations from the base model (e.g., LISA, EWC).
2. Gradient-based: Identifying and inhibiting specific harmful gradient directions (e.g., Booster, Gradient Route).
3. Representation-based: Constraining the drift of internal hidden representations (e.g., Vaccine, T-Vaccine).

The paper argues that these defenses consistently fail under persistent HFT. The authors attribute this failure to the inherent redundancy of the high-dimensional parameter space. Attackers can exploit optimization trajectories that are orthogonal to the specific constraints imposed by these defenses, effectively bypassing safety barriers while satisfying the imposed restrictions.

2. Methodology: Safety Bottleneck Regularization (SBR)

To address the limitations of constraining the redundant parameter space, the authors propose Safety Bottleneck Regularization (SBR). This method shifts the defensive focus from the internal parameters to the unembedding layer, which the authors identify as a "geometric bottleneck."

Core Concept

In a Transformer-based LLM, the probability of generating a token $t$ is determined by the linear projection of the final hidden state $h_{final}$ onto the token's embedding vector $w_t$:
$$\text{Score}(t) = h_{final}^\top w_t$$
The authors posit that while internal parameters can evolve along infinite redundant trajectories, the generation of a specific token strictly requires the final hidden state to align with that token's embedding direction. Therefore, controlling the final hidden state acts as a deterministic bottleneck.

Algorithm

SBR operates in two phases:

1. Offline Anchor Acquisition: Before fine-tuning, the defender extracts the final hidden states ($h_{ref}$) for a set of high-risk queries ($X_{anchor}$, e.g., "How to make a bomb?") using the frozen, safety-aligned base model. These states serve as Safety Anchors.
2. Dynamic Regularization: During fine-tuning on the user's dataset ($D_{train}$), the model optimizes standard Cross-Entropy loss ($L_{CE}$) while simultaneously minimizing the Mean Squared Error (MSE) between the current model's final hidden states for the anchor queries and the cached anchors:
   $$L_{safe}(\theta) = \frac{1}{|X_{anchor}|} \sum_{x' \in X_{anchor}} \| h_\theta(x') - h_{ref}(x') \|_2^2$$
   The total objective is:
   $$L_{total}(\theta) = L_{CE}(D_{train}; \theta) + \lambda \cdot L_{safe}(\theta)$$

Theoretical Justification

The paper provides theoretical analysis (Appendix E) demonstrating that:

• Orthogonality: Randomly sampled attack subspaces are geometrically orthogonal to one another with high probability in high dimensions.
• Feasibility: A randomly selected subspace almost surely contains a valid descent direction for the harmful objective, allowing attackers to bypass specific gradient or parameter constraints.
• Bottleneck Necessity: Because the final hidden state must project onto specific embedding vectors to generate harmful tokens, anchoring this state prevents the restoration of harmful capabilities regardless of how internal parameters evolve.

3. Key Contributions

1. Empirical and Theoretical Analysis of Defense Failure: The authors demonstrate that existing defenses collapse under persistent HFT because they fail to account for parameter redundancy. They show that attackers can find orthogonal optimization paths that satisfy defense constraints (e.g., parameter distance limits) while successfully restoring harmful capabilities.
2. Proposal of SBR: A novel defense mechanism that anchors the final hidden states of high-risk queries to those of the safety-aligned model, leveraging the unembedding layer as a geometric bottleneck.
3. High Efficiency and Robustness: The paper demonstrates that SBR requires minimal resources (as few as a single safety anchor) to maintain robust safety, outperforming state-of-the-art defenses while preserving utility on benign tasks.

4. Experimental Results

Experiments were conducted on Llama3.1-8B, Qwen2.5-7B, and Gemma1.1-7B using datasets like BeaverTails (harmful) and SST-2/AGNEWS/GSM8K/AlpacaEval (benign).

• Safety Performance: Under persistent HFT (up to 50 epochs), existing defenses (LISA, Vaccine, Booster) collapsed, with Harmful Scores (HS) rising above 50-60. In contrast, SBR maintained an average HS of 5.68 (significantly lower than the baseline SFT's 70.7).
• Utility Preservation: SBR preserved Functional Accuracy (FA) comparable to unconstrained fine-tuning (e.g., 94.15% on SST-2), whereas some competing methods (like DeepAlign) suffered performance degradation on classification tasks.
• Robustness to Poison Ratio: SBR remained effective even when the proportion of harmful data in the training set increased to 30% ($p=0.3$), maintaining HS < 10 across all model architectures.
• Backdoor Defense: SBR demonstrated superior robustness against stealthy backdoor jailbreaking attacks (triggered by specific suffixes), achieving an HS of 4.7 compared to >16 for other defenses.
• Anchor Efficiency: The method is agnostic to specific anchor content; a single anchor ($|X_{anchor}|=1$) was sufficient to reduce HS from ~68 to ~6.

5. Significance and Claims

The paper claims that SBR represents a paradigm shift in safety alignment, moving from constraining the vast, redundant internal parameter space to enforcing constraints at the deterministic geometric bottleneck of the unembedding layer.

• Fundamental Mechanism: The authors argue that safety collapse is not merely a result of parameter drift but a geometric misalignment at the final output layer. By anchoring this layer, SBR creates a "local stability field" that covers the semantic neighborhood of malicious intents without overfitting to specific examples.
• Orthogonality of Refusal: The method relies on the finding that refusal behaviors are mediated by subspaces largely orthogonal to benign reasoning. Consequently, anchoring safety states causes minimal interference with parameters required for downstream benign tasks.
• Limitations: The authors modestly note that SBR relies on the initial safety alignment of the base model and is designed to anchor existing boundaries rather than induce safety in non-aligned models. Furthermore, it does not directly mitigate inference-time jailbreak attacks that do not involve parameter updates.

The work concludes that defending at the geometric bottleneck is a necessary and effective strategy to prevent the removal of safety alignment during fine-tuning, offering a robust solution against both explicit and stealthy harmful fine-tuning attacks.