Technical Summary: From Parameter Dynamics to Risk Scoring

Problem Statement

Large Language Models (LLMs) rely on safety alignment (e.g., via RLHF or DPO) to refuse harmful requests. However, recent studies indicate that this alignment is fragile; fine-tuning on even a small number of benign samples can severely degrade model safety. Unlike overtly harmful data, which can be filtered by toxicity detectors, benign samples that induce safety degradation are difficult to identify and prevent. Existing research attempts to explain this phenomenon by comparing pre- and post-fine-tuning parameters or analyzing embedding drift. However, these approaches suffer from two primary limitations:

1. Static Analysis: They fail to capture the dynamic evolution of parameters during the fine-tuning process.
2. Directional Blindness: They focus on the magnitude of perturbations without considering the specific directions of parameter changes that drive safety degradation, making it difficult to isolate safety-specific risks.

Consequently, there is a lack of a method to quantify the risk of individual benign training samples in a way that explains why they degrade safety and allows for continuous risk assessment across the entire corpus.

Methodology

1. Mechanism Discovery: Parameter Dynamics

The authors investigate the mechanism of safety degradation by tracking parameter trajectories throughout the fine-tuning process rather than just comparing endpoints.

• Direction Definition: They define two reference directions in parameter space:
  • Safety Direction ($V_{safety}$): The parameter displacement from a base model to a safety-aligned state (via DPO on PKU-SafeRLHF).
  • Danger Direction ($V_{danger}$): The parameter displacement from a base model to a harm-aligned state (via SFT on unsafe datasets like Aegis and Beaver-Tails).
• Dynamic Observation: By projecting cumulative parameter drift ($\Delta \theta_t$) onto these directions during training, the authors observe a critical mechanism: benign fine-tuning induces a cumulative drift toward danger-aligned directions while the projection onto the safety direction remains near zero.
• Nonlinear Degradation: This drift exhibits a two-phase pattern. Initially, parameters drift rapidly in the danger direction with moderate safety loss. Subsequently, even as the drift rate slows, safety collapses catastrophically once the parameters exit the "safe basin."

2. Proposed Method: Sample-Level Quantification of Safety Degradation (SQSD)

Based on the finding that samples contributing more to the drift toward danger directions pose higher risks, the authors propose SQSD.

• Core Concept: SQSD quantifies the risk of a single training sample $z$ by measuring the projection gap between the sample-induced parameter update and the safety vs. danger directions.
• Computation Steps:
  1. Update Calculation: Compute the one-step gradient descent update ($\Delta W$) induced by sample $z$ on LoRA parameters.
  2. Normalization: Apply module-wise normalization to the update to mitigate response-length bias (a known issue in gradient-based scoring where short responses yield inflated gradients).
  3. Projection: Project the normalized update onto the danger direction ($\hat{V}_{danger}$) and safety direction ($\hat{V}_{safety}$).
  4. Aggregation: Calculate the score as the difference between these projections:
     $$SQSD(z) = \sum_m \left( \langle \frac{\Delta W_m}{\|\Delta W_m\|}, \hat{V}_{danger, m} \rangle - \langle \frac{\Delta W_m}{\|\Delta W_m\|}, \hat{V}_{safety, m} \rangle \right)$$
     A higher score indicates the sample pushes parameters more strongly toward the danger state than the safety state.
• Theoretical Basis: The authors link SQSD to output preferences using a first-order Taylor approximation. They demonstrate that the inner product of the update and a direction correlates with the change in loss (and thus likelihood) for that direction. Thus, a high SQSD implies the sample increases the model's preference for dangerous outputs relative to safe ones.
• Initialization Sensitivity: The method acknowledges that risk quantification depends on the initialization state. To ensure reliability, SQSD is computed at parameter states exhibiting high directional sensitivity (identified via drift-enhanced sensitivity analysis).

Key Contributions

1. Mechanistic Insight: The paper reveals that safety degradation in benign fine-tuning is driven by the cumulative drift of parameters toward danger-aligned directions, a dynamic process overlooked by static pre/post analyses.
2. SQSD Method: A novel method for continuous, sample-level risk quantification. Unlike prior works that identify discrete "high-risk" subsets (leading to boundary collapse), SQSD provides a risk score for every sample across the entire spectrum.
3. Empirical Validation: Extensive experiments across three models (Qwen3-8B, Llama-3.1-8B-Instruct, Llama-2-7B-Chat) and two datasets (Dolly, Alpaca) demonstrate that SQSD effectively predicts safety degradation.
4. Transferability: SQSD scores computed on one configuration (model architecture, parameter scale, or fine-tuning method like LoRA) transfer robustly to others, enabling risk assessment on smaller models or different architectures before deployment.

Results

• Effectiveness: When datasets are partitioned into subsets based on SQSD risk scores (S1: highest risk to S5: lowest risk), models fine-tuned on S1 exhibit significantly higher Attack Success Rates (ASR) and lower Safety Scores compared to S5.
  • SQSD achieved monotonic safety degradation (decreasing ASR from S1 to S5) in 10 out of 12 configurations.
  • In contrast, baselines (Reward Models, Bi-Anchor, Self-Inf-N, LARF) failed to maintain monotonicity in the majority of cases.
  • SQSD demonstrated superior discriminative power, with an average ASR difference ($\Delta$) of 49.86% between the highest and lowest risk subsets, significantly outperforming the best baseline (43.76%).
• Transferability:
  • Architecture: Scores transferred effectively between Llama and Qwen architectures.
  • Scale: Scores computed on 8B models successfully predicted risks for 14B and 32B models.
  • Method: Scores derived from LoRA gradients remained effective when applied to full fine-tuning scenarios.
• Ablation Studies: Removing module-wise normalization caused severe performance degradation due to response-length bias. Using only danger or only safety directions also failed to maintain monotonicity, confirming the necessity of contrasting both directions.

Significance and Claims

The paper claims that SQSD provides a defensive tool for model developers. By identifying benign samples that disproportionately drive safety degradation, practitioners can curate training data to preserve safety alignment during downstream adaptation.

The authors emphasize that while the technique could theoretically be misused to select data that maximally degrades safety, the primary goal is defensive. They advocate for the integration of SQSD into safety fine-tuning algorithms to create "risk-aware" training pipelines that adapt to downstream tasks without compromising safety. The work highlights that safety alignment is not a static property but a dynamic state vulnerable to specific directional drifts, and that quantifying these drifts at the sample level is essential for robust LLM deployment.