Technical Summary: Fine-Tuning Without Forgetting via Loss-Adaptive Learning Rates

Problem Statement

Fine-tuning large language models (LLMs) on new, task-specific corpora often leads to catastrophic forgetting, where capabilities acquired during pretraining (such as general reasoning, safety alignment, and factuality) degrade significantly. Existing mitigation strategies generally fall into two categories:

1. Data Replay: Mixing pretraining data with task data during fine-tuning. This is often infeasible as pretraining data for modern LLMs is proprietary or unavailable.
2. Objective Modification: Suppressing the influence of "hard" tokens or sequences (those assigned high loss by the pretrained model) during fine-tuning. Methods include downweighting high-loss sequences (FLOW), masking high-entropy tokens (STM), or reweighting losses (TALR, DFT).

The paper identifies a critical flaw in objective-modification approaches: in scenarios requiring the acquisition of new knowledge (e.g., learning new facts) or low-resource language adaptation, the "hard" tokens are precisely those essential for learning the new task. Suppressing them improves forgetting metrics but severely compromises target-task performance, creating a poor Pareto trade-off.

Methodology: FINCH

The authors propose FINCH (Forgetting-aware Inverse loss schedule), a method that controls forgetting not by altering the training objective or data distribution, but by adapting the learning rate on a per-step basis.

Theoretical Foundation

The core insight is a theoretical bound on per-step forgetting. The authors derive that the increase in loss on old tasks ($\Delta L_{old}$) at any training step $i$ is bounded by the product of the learning rate ($\eta_i$) and the square root of the current mini-batch loss ($L_{B_i}$):
$$\Delta L_{old}(p_i) \leq C_1 \eta_i \sqrt{L_{B_i}(\theta_i)} + C_2 \eta_i^2 L_{B_i}(\theta_i)$$
This implies that steps with high training loss (typically occurring early in fine-tuning or on difficult samples) are the primary drivers of forgetting.

The Adaptive Schedule

To keep the per-step forgetting bound uniform throughout training, FINCH sets the learning rate inversely proportional to the square root of the mini-batch loss:
$$\eta_i = \frac{\kappa}{\sqrt{L_{B_i}(\theta_i)}}$$
In practice, this is implemented using an exponential moving average (EMA) of the mini-batch losses ($\bar{L}_i$) to ensure stability, with a cap on the maximum learning rate ($\eta_{max}$) and a small epsilon for numerical stability:
$$\eta_i = \min\left( \frac{\eta_{base}}{\sqrt{\bar{L}_i + \epsilon}}, \eta_{max} \right)$$
Key Mechanism:

• High-Loss Batches: The learning rate is reduced, minimizing the magnitude of updates that could damage retained capabilities.
• Low-Loss Batches: As the model converges and loss decreases, the learning rate increases, allowing for efficient optimization.
• Objective Preservation: Unlike other methods, FINCH leaves the training objective (cross-entropy) unchanged. High-loss tokens still contribute to learning, ensuring the model can acquire new knowledge.

Key Contributions

1. Theoretical Bound: The paper establishes a theoretical link between per-step forgetting, learning rate, and training loss, demonstrating that forgetting is bounded by $\eta \sqrt{L}$.
2. FINCH Algorithm: A simple, data-oblivious learning rate schedule that requires no access to pretraining data and no modification of the fine-tuning objective.
3. Pareto Optimization: FINCH achieves a superior trade-off between target-task accuracy and the preservation of general capabilities compared to state-of-the-art baselines.

Experimental Results

The authors evaluate FINCH on three settings where pretrained models have limited coverage: Knowledge Acquisition (TOFU), Science Reasoning (SciKnowEval), and Low-Resource Language Adaptation (Galician Alpaca). Experiments are conducted on Qwen3-4B and Llama-3-8B.

Performance Metrics

• Forgetting Reduction: FINCH reduces average forgetting by 93% compared to standard Supervised Fine-Tuning (SFT) across all tasks.
• Target Task Accuracy: FINCH matches or exceeds the target-task accuracy of standard SFT and LoRA. For example, on Knowledge Acquisition with Qwen3-4B, FINCH achieves 83.3% accuracy (comparable to SFT's 83.5%) while reducing average benchmark degradation from -9.6 to +0.1.
• Reliability Preservation:
  • Factuality: On TruthfulQA, FINCH reduces degradation by 5$\times$ compared to SFT (from -13.9 to -2.6 points).
  • Hallucination Detection: On HaluEval, FINCH reverses the degradation seen in SFT, improving performance by +3.3 points.
  • Calibration: FINCH better preserves confidence calibration (measured by Brier score) than standard SFT, though some degradation remains.

Comparison with Baselines

• Objective-Modification Methods (FLOW, DFT, STM): These methods often reduce forgetting but fail to learn the new task effectively, resulting in significantly lower target-task accuracy.
• Parameter-Efficient Methods (LoRA): While effective at reducing forgetting, LoRA still incurs significant degradation in general capabilities compared to FINCH.
• Fixed Small Learning Rate: Simply lowering the learning rate reduces forgetting but fails to reach competitive new-task accuracy, confirming that the adaptive nature of FINCH is crucial.

Significance and Claims

The paper claims that learning-rate schedules are an underutilized tool for shaping model behavior during fine-tuning. By dynamically adjusting the step size based on the difficulty of the current batch (as reflected by loss), FINCH allows models to learn new, difficult information without catastrophically overwriting pretraining knowledge.

The authors emphasize that their approach is data-oblivious, making it practical for modern LLMs where pretraining data is inaccessible. They conclude that FINCH provides a Pareto-optimal solution, achieving high target-task performance while simultaneously preserving factuality, hallucination detection, and general reasoning capabilities, outperforming methods that attempt to mitigate forgetting by suppressing hard examples.

Limitations Acknowledged:

• The theoretical analysis is derived for SGD, though experiments use AdamW.
• While FINCH reduces calibration degradation, it does not fully preserve the pretrained model's calibration.
• Experiments are limited to models up to 8B parameters due to computational constraints.