Alignment Dynamics in LLM Fine-Tuning

Technical Summary: Alignment Dynamics in LLM Fine-Tuning

Problem Statement
Large Language Models (LLMs) achieve alignment with human values through supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF). However, this alignment is often fragile; models can rapidly lose safety guardrails or revert to pre-trained behaviors when subjected to subsequent fine-tuning, even on benign data. Existing explanations for this fragility fall into two categories: gradient-based analyses (focusing on geometry in parameter space) and distribution-based studies (focusing on shifts in output distributions). A critical gap remains in providing a unified theoretical framework that bridges parameter-space learning dynamics with function-space alignment behavior, specifically explaining how alignment evolves, degrades, and recovers during fine-tuning.

Methodology
The authors introduce a tractable alignment score $S(\theta)$, defined as the expected probability of generating an aligned completion given a prompt distribution. They derive a closed-form expression for the evolution of this score during fine-tuning.

1. Theoretical Framework:

   • Token-Level Decomposition: The change in the alignment score ($\Delta S$) is decomposed into contributions from individual token updates. This decomposition reveals that the contribution of a token update is "gated" by the prefix probability and the "future alignment potential" (the probability that the completion will eventually be aligned).
   • Logit-Space Geometry: By connecting these token-level dynamics to logit perturbations via the empirical Neural Tangent Kernel (eNTK) and the Relatively Stable Kernel (RSK) assumption, the authors derive a unified formula for alignment dynamics.
   • Force Decomposition: The evolution of the alignment score is shown to be governed by two competing forces:
     • Rebound Force ($F_{rebound}$): An intrinsic self-interaction term representing "distributional inertia." It is modulated by the current alignment state and the narrowness (purity) of the model's posterior distribution.
     • Driving Force ($F_{drive}$): An external term determined by how the training distribution aligns with the outcome-conditioned posteriors over aligned ($\tilde{\pi}^+$) and non-aligned ($\tilde{\pi}^-$) completions.

2. Experimental Validation:

   • The authors employ a multi-stage SFT paradigm (Forward $\to$ Reverse $\to$ Re-exposure) across three domains: safety alignment (BeaverTails, HEx-PHI), emergent misalignment, and sentiment analysis (IMDb).
   • They utilize base models (Llama-3.1-8B, Gemma-2-2B, Qwen3-8B-Base) to isolate alignment dynamics from prior instruction tuning.
   • Controlled experiments vary the narrowness of the fine-tuning distribution (from fixed templates to style-diversified responses) to test theoretical predictions regarding the Rebound Force.

Key Contributions and Results

1. Unified Dynamical Framework: The paper provides a mathematical derivation showing that alignment dynamics are a competition between internal resistance (Rebound Force) and external learning signals (Driving Force). This explains why alignment can be reversed by subsequent fine-tuning and why the structure of the posterior distribution matters.
2. Bidirectional Instability and Distribution Narrowness:
   • The framework predicts that both strongly aligned and strongly misaligned states are "unstable" and prone to reversion.
   • Result: Experiments confirm that narrower posterior distributions (induced by low-diversity fine-tuning data) amplify the Rebound Force. Models trained on narrow datasets exhibit faster degradation of safety alignment when exposed to reverse fine-tuning compared to those trained on diverse datasets.
3. The Rehearsal Priming Effect:
   • Prediction: Prior alignment leaves a latent "posterior imprint" (structural residue in $\tilde{\pi}^+$ and $\tilde{\pi}^-$) that persists even after the observed alignment score is suppressed. Upon re-exposure to the original alignment distribution, this residue amplifies the effective Driving Force, leading to faster re-alignment.
   • Result: Controlled experiments demonstrate that models which underwent deeper initial fine-tuning (Stage 1) realign significantly faster during Stage 3 (Re-exposure) compared to those with shallower initial training, even when starting from the same baseline alignment score. This effect holds across safety, misalignment, and sentiment tasks.
4. Emergent Misalignment Fragility: The study highlights that models exhibiting emergent misalignment (broad misaligned behaviors induced by narrow fine-tuning) can have these behaviors rapidly reactivated with only a few steps of malicious fine-tuning, underscoring the inherent fragility of subsequent alignment.

Significance and Claims
The authors claim that this work offers a unified dynamical perspective on how alignment is disrupted and reactivated during LLM fine-tuning. By moving beyond static distributional shifts or purely geometric gradient analyses, the proposed framework connects token-level optimization dynamics with global alignment behavior.

The significance of the work lies in:

• Providing a rigorous, quantitative characterization of alignment evolution.
• Explaining the mechanism behind the "Rebound Effect" and identifying distribution narrowness as a key modulator of alignment stability.
• Discovering and validating the Rehearsal Priming Effect, suggesting that alignment robustness is not solely determined by the training signal but also by the latent posterior structure induced by prior training.
• Offering a principled foundation for analyzing and potentially improving the robustness of post-training alignment, while acknowledging that these insights could also inform adversarial attacks on alignment.

The paper concludes that alignment is governed by the interplay between the training signal and the induced posterior structure, a finding that challenges the view of alignment as a static property and reframes it as a dynamic process susceptible to specific structural vulnerabilities.