RefusalGuard: Geometry-Preserving Fine-Tuning for Safety in LLMs

Problem Statement

Large Language Models (LLMs) are typically safety-aligned via instruction tuning and preference optimization. However, when these models undergo post-alignment fine-tuning for downstream tasks (e.g., domain specialization or user preferences), they often suffer a substantial degradation in refusal behavior. This fragility persists even when the fine-tuning data is benign or when the model is exposed to only a minimal number of harmful examples (e.g., 10 synthetic harmful prompts).

While prior work has established that alignment can be brittle under such adaptation, the internal mechanisms driving this failure remain poorly understood. Existing mitigation strategies often rely on heuristic prompt design, additional safety examples, or adversarial training without a principled understanding of what internal structures must be preserved. The paper posits that the loss of safety is not merely generic catastrophic forgetting but a specific geometric failure within the model's activation space.

Methodology

1. Mechanistic Analysis of Alignment Degradation

The authors first investigate the representation-level mechanisms underlying safety loss using the lens of refusal geometry. Building on prior work suggesting that refusal is mediated by structured, low-dimensional geometric structures (cones) in the residual stream, the paper analyzes how these structures evolve during standard fine-tuning.

The analysis identifies three consistent effects that accompany safety degradation:

• Directional Drift: Harmful-input representations move away from the reference refusal cone defined by the original aligned model. The magnitude of activation within the refusal subspace decreases, and the cosine similarity between activations and the refusal basis drops.
• Geometric Distortion: The refusal geometry itself changes. The subspace spanned by the refusal cone rotates (drifts) relative to the original, and the internal distribution of coordinates within the cone becomes concentrated and unstable (reduced entropy, increased top-1 mass), making refusal brittle.
• Task-Safety Interference: Optimization updates for downstream tasks are not orthogonal to safety-relevant directions. Instead, a significant portion of the gradient updates for harmful prompts lies parallel to the refusal subspace, actively modifying and destabilizing the safety-mediating components.

2. RefusalGuard Framework

Motivated by the finding that safety loss stems from the displacement and distortion of refusal geometry, the authors propose RefusalGuard, a representation-level fine-tuning framework.

• Core Paradigm: The method utilizes Representation Finetuning (ReFT), where the base model parameters are frozen, and low-rank intervention modules are learned to modify hidden states during the forward pass.
• Geometric Constraint: The key innovation is a geometry-preservation loss ($L_{geom}$). Before fine-tuning, a low-dimensional refusal cone is extracted from the aligned base model to define a refusal-relevant subspace. During training, the intervention updates are penalized if they have a large component within this subspace.
  • The hidden state update $\delta$ is projected onto the refusal subspace ($P_{ref}$).
  • The loss minimizes the norm of this projection: $L_{geom} = \mathbb{E}[\|P_{ref}(\tilde{h} - h)\|_2^2]$.
• Objective: The total training objective combines the standard downstream task loss ($L_{task}$) with the geometry-preservation loss:
  $$L_{RG} = L_{task} + \lambda_{geom} L_{geom}$$
  This ensures that task-specific learning proceeds in directions complementary to the refusal geometry, while the safety-mediating components remain stable.

Key Contributions

1. Representation-Level Analysis: The paper provides a mechanistic account of alignment degradation, demonstrating that it is driven by systematic directional drift, geometric distortion of refusal cones, and interference between task optimization and safety features.
2. RefusalGuard Framework: It introduces a training method that explicitly preserves the geometric structure of refusal during adaptation, rather than relying solely on output-level supervision or prompt engineering.
3. Empirical Validation: The approach is evaluated across multiple model families (LLaMA, Gemma, Qwen) and benchmarks, showing that preserving refusal geometry effectively prevents safety collapse while maintaining competitive utility.

Experimental Results

The authors evaluate RefusalGuard under a stress-test setting where aligned models are fine-tuned on only 10 harmful synthetic examples.

• Safety Performance:
  • Baseline Failure: Standard LoRA fine-tuning causes a catastrophic drop in safety, with Attack Success Rates (ASR) rising to ~0.70–0.75 and refusal rates dropping to ~0.25–0.30 across benchmarks (AdvBench, DirectHarm4, JailbreakBench).
  • Comparison: Representation-level baselines like Circuit Breaker (CB) and REPBEND improve robustness but remain significantly behind RefusalGuard (ASR ~0.10–0.27).
  • RefusalGuard: Achieves near-zero ASR (0.008–0.03) and maintains refusal rates above 0.97, effectively preventing the collapse induced by harmful adaptation.
• Utility Performance:
  • While RefusalGuard incurs a modest reduction in downstream utility (e.g., GSM8K, OpenOrca, ARC) compared to unconstrained LoRA, the retained performance remains competitive. The paper notes a favorable safety-utility trade-off, where significant safety gains are achieved with only minor capability losses.
• Mechanistic Correlation: The experimental results confirm the mechanistic hypothesis: models trained with RefusalGuard maintain high alignment with the original refusal geometry, low cone drift, and minimal task-safety interference, whereas standard fine-tuning exhibits the opposite trends.

Significance and Claims

The paper claims that the fragility of safety-aligned models under fine-tuning is not an inevitable consequence of adaptation but a specific geometric failure. By identifying that refusal is mediated by identifiable, low-dimensional structures, the authors argue that safety can be preserved by constraining updates to avoid displacing these structures.

The significance of RefusalGuard lies in its shift from heuristic defenses to a principled, representation-level approach. It demonstrates that preserving the internal geometric integrity of refusal is a viable strategy for robust model adaptation. The authors conclude that this approach offers a more mechanistically grounded solution to maintaining safety in downstream applications, particularly in scenarios where models must be specialized without compromising their alignment.