🤖 machine learning

(How) Learning Rates Regulate Catastrophic Overtraining

This paper investigates catastrophic overtraining in LLMs during supervised fine-tuning, revealing that learning rate decay increases model sharpness and exacerbates catastrophic forgetting, thereby demonstrating how learning rates mediate optimization dynamics to produce qualitatively different models.

Original authors: Mark Rofin, Aditya Varre, Nicolas Flammarion

Published 2026-04-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mark Rofin, Aditya Varre, Nicolas Flammarion

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, well-traveled librarian named Alex. Alex has spent years reading millions of books (this is Pretraining). As a result, Alex knows a little bit about everything: history, science, cooking, and how to tell a good joke. Alex is a "generalist."

Now, you want to hire Alex to work at a specific, strict library branch where the only job is to answer customer questions politely and follow a very specific format (this is Supervised Finetuning or SFT).

The problem? When you train Alex for this new job, something strange happens. If you train Alex too hard or too fast, Alex forgets how to be a generalist. Alex stops knowing about history or science and becomes a robot that only knows how to answer questions in the new format, often giving terrible answers to anything outside that format. This is called Catastrophic Forgetting.

Even worse, the paper discovers a new twist: Catastrophic Overtraining. The more you train Alex on the general knowledge (the more books Alex reads before the new job), the more Alex forgets everything else when you finally try to teach the new job. It seems counterintuitive: more training should make Alex smarter, but here, it makes Alex more fragile.

This paper asks: Why does this happen, and how do we fix it?

The answer lies in the Learning Rate, which is essentially the "pace" at which Alex learns the new job.

The Two Paces of Learning

The authors found that the speed at which you teach Alex the new job changes the outcome completely.

1. The "Sprint" (High Learning Rate)
Imagine you tell Alex: "Forget everything you know! Learn this new format immediately, and learn it FAST!"

What happens: Alex panics. To learn the new format quickly, Alex has to completely rewrite their brain. They discard old memories (history, science) to make room for the new rules.
The Result: Alex becomes great at the new format but loses all their general knowledge. They are "sharp" and rigid. If you ask them a question about history, they can't answer.

2. The "Stroll" (Low Learning Rate)
Now, imagine you tell Alex: "Take your time. Learn the new format, but don't throw away your old knowledge. Just gently adjust your habits."

What happens: Alex learns the new format slowly. Because the changes are small, Alex doesn't need to delete old memories to make space. They keep their general knowledge intact while slowly adopting the new rules.
The Result: Alex is good at the new format and still remembers history and science. They are "soft" and flexible.

The Hidden Culprit: "Sharpness"

The paper introduces a concept called Sharpness. Think of a Loss Landscape as a mountain range where the bottom of the valley is the perfect answer.

A "Flat" Valley: The bottom is wide and gentle. If you take a step, you don't fall off the edge easily. This is stable.
A "Sharp" Peak: The bottom is a tiny, needle-like point. If you take even a tiny step, you might roll right off the side. This is unstable.

The paper discovered that as Alex reads more books (Pretraining), their brain naturally becomes "sharper" (more like a needle). This happens because the training process usually slows down (the Learning Rate decays) as Alex gets smarter.

The Trap:
When Alex is already standing on a "sharp needle" (a highly trained base model) and you try to teach them the new job with a fast pace (High Learning Rate), they are like a tightrope walker trying to run. They stumble, fall off the needle, and completely lose their balance (forgetting everything).

However, if you teach them with a slow pace (Low Learning Rate), they can carefully walk along the needle without falling off. They preserve their balance (their general knowledge).

The Big Discovery: The "Overtraining" Paradox

The paper explains the scary phenomenon of Catastrophic Overtraining:

You train Alex on more and more books (Pretraining).
This makes Alex's brain "sharper" (more fragile).
When you finally try to teach the new job, if you use a standard (fast) pace, Alex's sharp brain makes them fall off the cliff immediately.
Conclusion: The more you pretrain, the more "sharp" and fragile the model becomes, leading to more forgetting during the new training.

The Solution: Slow Down and Soften

The paper offers two practical pieces of advice for building better AI:

During the New Job (Finetuning): Use a very slow learning rate. Don't rush the model. Let it learn the new rules gently so it doesn't forget its old skills. Think of it as "gentle retraining" rather than "hard drilling."
During the Old Job (Pretraining): Don't slow down the learning rate too much at the end. If you keep the pace steady, the model stays "flatter" and more robust, making it easier to teach new things later without forgetting.

Summary Analogy

Imagine you are sculpting a statue out of clay.

Pretraining is shaping the clay into a rough human form.
Finetuning is carving the fine details (like a specific facial expression).

If the clay has become rock-hard (too much pretraining/sharpness) and you try to carve the details with a heavy chisel (high learning rate), you will crack the whole statue. The details will be perfect, but the statue will fall apart.

But if you use a fine, gentle tool (low learning rate), you can carve the details without breaking the statue. The paper proves that the "hardness" of the clay increases the more you work on it initially, so you must be extra gentle when you start the final details.

1. Problem Statement

The paper addresses Catastrophic Overtraining in Large Language Models (LLMs), a phenomenon recently identified where prolonged pretraining leads to worse performance after Supervised Finetuning (SFT), despite the base model having better pretraining performance.

Context: LLMs are typically trained via pretraining (unsupervised) followed by SFT (instruction-following).
The Issue: While SFT is necessary for alignment, it often causes Catastrophic Forgetting, where the model loses general capabilities (Out-of-Distribution or OOD performance).
The Paradox: Recent work (Springer et al., 2025) showed that increasing the pretraining budget (more tokens/steps) actually increases this forgetting. A model pretrained for longer ( $t_2$ ) performs worse on OOD tasks after SFT than a model pretrained for less time ( $t_1$ ), even though the longer-pretrained model was better before SFT.
Hypothesis Gap: Previous explanations suggested this was due to "progressive sensitivity" or capacity limits. However, recent findings suggest optimization dynamics (specifically Learning Rates) play a critical role, which this paper aims to mechanistically explain.

2. Methodology

The authors investigate the interplay between Learning Rates (LR) in both pretraining and finetuning, and the concept of Loss Landscape Sharpness.

Experimental Setup:
- Models: OLMo-1B, OLMo-2-1B, Hubble-1B, Gemma-3-1B, and SmolLM-3B.
- Datasets: SFT on Anthropic-HH (instruction following) and OOD evaluation on 6 standard tasks (ARC, PIQA, etc.).
- Variables:
  - Finetuning LR: Tested a range of constant LRs (from $10^{-6}$ to $10^{-4}$ ) to reach the same final SFT loss.
  - Pretraining Checkpoints: Selected checkpoints at different stages of pretraining (early vs. late) to vary the "sharpness" of the base model.
Metrics:
- OOD Score: Average accuracy on general tasks to measure forgetting.
- Mean Principal Angle (MPA): A metric quantifying the drift in residual stream representations between the base model and the finetuned model. Higher MPA indicates greater feature drift.
- Sharpness Proxy: Since computing the Hessian eigenvalue is infeasible for large models, they use a perturbation-based proxy: the KL-divergence between the original model's predictions and those of a model with Gaussian noise added to parameters.
Theoretical Analysis:
- Used a Two-Layer Diagonal Network (sparse regression) to mathematically model the trade-off between preserving pretrained features and learning new SFT features under different LRs.
- Analyzed the Loss Landscape by interpolating between optimization steps to observe how loss and feature drift evolve.

3. Key Contributions & Findings

A. Learning Rate as Implicit Regularization (Finetuning)

Finding: For a fixed SFT loss, models finetuned with lower Learning Rates retain significantly better OOD capabilities than those finetuned with high LRs.
Mechanism:
- High LR: Operates in a "rich" regime, causing rapid feature learning and large deviations from the initialization. This leads to a large Mean Principal Angle (MPA), meaning the model's internal representations drift far from the pretrained state, causing forgetting.
- Low LR: Operates in a "lazy" regime. The model stays closer to the initialization, preserving pretrained features while still fitting the SFT data.
Conclusion: Low LRs implicitly regularize the model toward preserving base capabilities.

B. The Role of Sharpness

Finding: The "sharpness" of the loss landscape (related to the curvature/Hessian) dictates how sensitive the model is to optimization steps.
Mechanism:
- In a sharper region of the loss landscape, a given step size causes a larger change in the model's features (higher MPA) relative to the loss reduction.
- The authors demonstrate that increasing the LR has an effect mathematically similar to increasing the sharpness of the initialization: both accelerate feature drift.

C. The Root Cause of Catastrophic Overtraining

Finding: The phenomenon of catastrophic overtraining is driven by Learning Rate Decay during Pretraining.
Causal Chain:
1. Standard pretraining schedules use Learning Rate Decay (e.g., Warmup-Stable-Decay).
2. As the LR decays, the model enters the Edge of Stability regime, causing the Sharpness of the loss landscape to increase (the model settles into sharper minima).
3. A sharper base model is more prone to feature drift when finetuned with a fixed LR.
4. Consequently, later pretraining checkpoints (which are sharper) suffer more catastrophic forgetting during SFT than earlier, flatter checkpoints.
Evidence: In models using WSD (Warmup-Stable-Decay) schedules, the spike in forgetting and feature drift (MPA) coincides exactly with the phase where the pretraining LR is decayed and sharpness increases.

D. Theoretical Validation

Using a diagonal network model, the authors proved that large LRs drive the model to overwrite pretrained features (features specific to pretraining but not SFT), while small LRs allow the model to fit the new task without erasing the old ones. This confirms the "lazy vs. rich" regime hypothesis.

4. Results Summary

Figure 1: Shows that for the same SFT loss, higher finetuning LRs result in lower OOD scores (more forgetting).
Figure 2 & 3: Demonstrates that MPA (feature drift) increases linearly with LR, while loss decreases convexly. High LRs cause the model to "overshoot" the preservation of features.
Figure 5: Shows that model sharpness increases dramatically during the LR decay phase of pretraining.
Figure 6 & 7: Confirms that as pretraining progresses (and LR decays), the resulting base models become sharper and exhibit significantly higher forgetting and feature drift when finetuned.

5. Significance and Implications

This paper provides a mechanistic explanation for why "more pretraining" can sometimes be detrimental to downstream performance, shifting the blame from simple capacity limits to optimization dynamics.

Practical Recommendations:
1. Finetuning: Use the smallest possible Learning Rate that achieves acceptable in-distribution performance to minimize feature drift and forgetting.
2. Pretraining: Avoid aggressive Learning Rate Decay during the pretraining phase. The authors suggest maintaining a stable LR or only annealing during the post-training (SFT/RL) phase to prevent the base model from becoming too sharp.
Theoretical Impact: It bridges the gap between the "Edge of Stability" theory, implicit regularization, and catastrophic forgetting, showing that the interaction between pretraining and finetuning hyperparameters is critical for model stability.

In summary, the paper argues that Catastrophic Overtraining is not an inevitable consequence of scaling data, but a side effect of standard LR schedules that increase model sharpness, making models fragile to subsequent finetuning.