Technical Summary: Stabilizing Supervised Fine-Tuning of LLMs via Explicit Distribution-Based Control

1. Problem Statement

Post-training Large Language Models (LLMs) to adapt to new objectives using Supervised Fine-Tuning (SFT) frequently leads to catastrophic forgetting, where improvements in a target task degrade previously acquired general capabilities. Recent empirical and theoretical studies suggest that this phenomenon is primarily caused by excessive distribution-based drift during optimization.

While Reinforcement Learning (RL) methods (particularly on-policy RL) implicitly constrain this drift and preserve prior capabilities better than SFT, they rely on costly data collection and complex optimization pipelines. In contrast, standard SFT lacks explicit mechanisms to regulate the magnitude or direction of distribution-based shifts, often leading to abrupt behavioral changes. Existing mitigation strategies, such as static KL-regularization or fixed teacher distillation, offer only indirect or globally tuned control that does not align with the local geometry of model distributions, frequently resulting in persistent gradient conflicts between adapting to the new task and preserving the base model.

The central challenge is: Can a simple, explicit mechanism enable stable control of distribution-based updates in a purely offline SFT setting, thereby combining the efficiency of SFT with the stability of on-policy RL?

2. Methodology: Anchored Learning

The authors propose Anchored Learning, a framework that explicitly controls distribution-based updates during offline fine-tuning via a dynamically evolving moving anchor.

Core Mechanism

Instead of adapting to a fixed reference distribution (e.g., the base model $p_{base}$), Anchored Learning constructs an intermediate target distribution $q^{(t)}$ at each iteration $t$ by interpolating between:

1. The current trainable model $p_{\theta}^{(t)}$.
2. A frozen SFT reference model $p_{sft}$ (trained once on target data and held fixed).

The framework operates in two loops:

• Outer Loop (Anchor Selection): At iteration $t$, an anchor distribution $q^{(t)}$ is constructed via an interpolation operator $I_\alpha$:
  $$q^{(t)}(\cdot | x) = I_\alpha(p_{\theta}^{(t)}(\cdot | x), p_{sft}(\cdot | x))$$
  The anchor is detached from the computation graph and serves as a fixed target for the inner loop.
• Inner Loop (Anchor Adaptation): Model parameters are updated to approximate this anchor by minimizing a distillation objective (KL-divergence) over the target dataset $D$:
  $$\theta^{(t+1)} \approx \arg \min_\theta \mathcal{L}_{distill}(\theta; q^{(t)})$$

Instantiations

The interpolation operator $I_\alpha$ is instantiated in two spaces:

1. Logit Space: Linear interpolation of pre-softmax logits, followed by softmax normalization. This corresponds to a geometric mean of distributions (behavior tending toward modes).
2. Probability Space: Linear interpolation of prediction distributions directly. This corresponds to an arithmetic mean (behavior covering modes).

Theoretical Guarantees

The work provides theoretical proofs that this anchor-based update allows for a linear upper bound on KL-divergence per iteration.

• Probability Space: The KL-divergence between the anchor and the current model is bounded by $\alpha \cdot KL(p_{sft} \parallel p_{\theta}^{(t)})$.
• Logit Space: The divergence is bounded by $\frac{\alpha}{1-\alpha} KL(p_{\theta}^{(t)} \parallel p_{sft})$.

These bounds ensure that the model transitions between distributions in a stable, controlled manner, effectively transforming global fine-tuning into a sequence of local trust-region updates. Theoretical analysis shows that under exact projection, the model converges geometrically toward the SFT reference while maintaining a bounded step-wise deviation.

3. Main Contributions

1. Framework Proposal: Introduction of Anchored Learning, a simple framework that enforces local control over distribution-based updates in offline fine-tuning via a moving anchor, thereby avoiding the need for online interaction or complex RL pipelines.
2. Theoretical Stability: Derivation of explicit KL-divergence upper bounds per iteration for both probability-space anchoring and logit-space anchoring, providing formal guarantees for distribution-based stability.
3. Empirical Validation: Demonstration that the method achieves a superior trade-off between performance gains and stability across multiple benchmarks, consistently positioning itself on the Pareto frontier compared to strong baselines.

4. Experimental Results

The method was evaluated on three target tasks (iGSM, MedCalc, IFEval) using Qwen2.5 (1.5B–14B) and Llama-3.2-3B models. Performance was measured against standard SFT and various forgetting-mitigation baselines (e.g., Low-SFT, KL-SFT, Self-SFT, Iter-SFT).

• Trade-off between Gain and Stability: Standard SFT achieved high domain performance but suffered from severe degradation of general capabilities (e.g., >53% performance drop on iGSM and MedCalc). Anchored Learning reduced this degradation to under 5% while maintaining near-optimal gains (e.g., 75.2% on iGSM).
• Pareto Optimality: Across all model sizes and tasks, Anchored Learning consistently positioned itself near the Pareto frontier, outperforming baselines that either sacrificed domain performance for stability (e.g., Low-SFT) or failed to prevent forgetting (e.g., Standard SFT).
• Scaling: The benefits of Anchored Learning were robust across model scales (1.5B to 7B), preserving significantly higher general capabilities than Low-SFT without sacrificing domain accuracy.
• Hyperparameter Sensitivity:
  • Interpolation Coefficient ($\alpha$): An intermediate range of $\alpha$ offered the best trade-off.
  • Update Schedule: The study highlighted that overall performance can degrade non-monotonically with increased training iterations, even when target accuracy saturates. This suggests that "more training" does not guarantee stability and underscores the need for explicit trajectory control.
  • Interpolation Space: Interpolation in the logit space (geometric mean) favored stability and preservation of general capabilities, while interpolation in the probability space (arithmetic mean) favored plasticity and domain adaptation.

5. Significance and Claims

The work claims that Anchored Learning offers a practical alternative to implicit stability mechanisms found in on-policy RL. By explicitly shaping intermediate target distributions, it transforms global optimization into a sequence of conservative, local updates.

The authors emphasize that their approach:

• Operates in a purely offline setting and requires no access to data from general domains for replay or joint training.
• Provides theoretical guarantees for distribution-based stability, addressing the root cause of forgetting (excessive drift) rather than just the symptoms.
• Achieves near-optimal performance on target tasks while significantly reducing catastrophic forgetting, thereby outperforming existing static regularization and distillation methods.

Limitations: The authors acknowledge that Anchored Learning incurs additional computational overhead compared to standard SFT (approximately 1.5 to 2 times training latency and memory usage) due to the need to construct anchors and perform additional forward passes for distillation. However, they note that this is significantly lower than the costs associated with on-policy RL.