🤖 machine learning

RL Fine-Tuning Heals OOD Forgetting in SFT

This paper challenges the notion that "SFT memorizes, RL generalizes" by demonstrating through checkpoint-wise and spectral analyses that SFT often causes out-of-distribution forgetting which is subsequently restored by RL, a recovery process linked to the rotation of singular vectors rather than changes in singular values.

Original authors: Hangzhan Jin, Sitao Luan, Tianwei Ni, Sicheng Lyu, Guillaume Rabusseau, Reihaneh Rabbany, Doina Precup, Mohammad Hamdaqa

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC0 1.0

Original authors: Hangzhan Jin, Sitao Luan, Tianwei Ni, Sicheng Lyu, Guillaume Rabusseau, Reihaneh Rabbany, Doina Precup, Mohammad Hamdaqa

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Two-Step" Dance of AI Training

Imagine you are teaching a brilliant but inexperienced student (a Large Language Model) how to solve complex puzzles. The standard recipe for making this student a reasoning expert involves two steps:

Step 1: Supervised Fine-Tuning (SFT) – You give the student a textbook with answers and say, "Memorize these patterns and copy the style."
Step 2: Reinforcement Learning (RL) – You put the student in a game where they get points for correct answers and lose points for wrong ones, saying, "Now, figure out the logic yourself to win."

For a long time, the AI community believed a simple story: "SFT makes the student memorize the textbook (good for known problems), and RL makes the student generalize and think creatively (good for new problems)."

This paper says that story is incomplete. The authors discovered that while SFT is great at first, it actually starts to hurt the student's ability to solve new types of puzzles if you let them study the textbook for too long. Then, RL doesn't necessarily teach them new superpowers; it mostly just fixes the damage SFT caused.

The Discovery: The "Peak and Crash"

The researchers watched the student's progress every single day during the training process. They found a surprising pattern:

The Early Win: At the very beginning of the "textbook study" (SFT), the student gets really good at solving new types of puzzles (Out-of-Distribution or OOD tasks). It's like the student suddenly understands the underlying logic of math.
The Crash: As the student keeps studying the textbook, they start to get worse at the new puzzles, even though they get better at the specific textbook problems. They become so obsessed with the exact format of the textbook answers that they forget how to apply the logic to slightly different situations.
The Fix: When they finally switch to the "game" phase (RL), the student's performance on new puzzles bounces back up.

The Analogy:
Imagine a chef learning to cook.

Early SFT: The chef learns the basic rules of flavor. They can cook a great meal with any ingredients.
Late SFT: The chef is forced to memorize a specific recipe book. They become amazing at making that exact dish but forget how to adjust flavors if they are given different ingredients. They have "forgotten" their general cooking intuition.
RL: The chef is put in a competition where they get points for tasty food, regardless of the recipe. This forces them to stop blindly following the book and start using their intuition again. They recover their general cooking skills, but they don't suddenly become a better chef than they were at the very start of their training.

The Key Finding: RL is a "Restorer," Not a "Creator"

The paper challenges the idea that RL creates new reasoning abilities from scratch. Instead, the authors found:

SFT Forgets: If you train too long on the specific data, the model "forgets" its ability to handle weird or new situations.
RL Recovers: RL acts like a bandage. It patches up the holes SFT made, restoring the model to the level of performance it had at the peak of the SFT phase.
The Ceiling: RL rarely makes the model better than that early peak. It just brings it back up to where it was before it got too specialized.

The "Goldilocks" Zone:
The researchers found that RL only works if you start it at the right time.

If you start RL too early (before the model knows the basics), it fails because the model is too confused.
If you start RL too late (after the model has forgotten everything), the "game" signals are too messy for the model to learn from.
There is a specific "sweet spot" (a range of checkpoints) where RL can successfully heal the forgetting.

The Secret Mechanism: The "Spinning Compass"

To understand why this happens, the authors looked inside the model's brain using a mathematical tool called Singular Value Decomposition (SVD). Think of the model's knowledge as a giant compass with many needles pointing in different directions.

The Size of the Needles (Singular Values): The researchers found that the strength of these needles (how much knowledge is stored) barely changes during training. They stay stable.
The Direction of the Needles (Singular Vectors): However, the direction the needles point to changes drastically.

The Analogy:
Imagine the model's knowledge is a map.

SFT doesn't erase the map (the size of the data stays the same), but it rotates the map. It turns the compass so that "North" points toward the specific textbook examples, making it hard to find "North" for new, different locations.
RL rotates the compass back. It doesn't add new land to the map; it just re-aligns the compass so "North" points to the general logic again, allowing the model to navigate new terrain.

Summary for the Everyday Reader

Old Belief: "SFT memorizes, RL generalizes."
New Reality: "SFT forgets (by over-specializing), and RL recovers (by re-aligning)."
The Lesson: Don't keep training your AI on the same specific data forever. It will lose its ability to think flexibly. If you want it to be smart about new things, you need to stop the "memorization" phase at the right time and use the "game" phase to fix the focus, rather than expecting the game to teach it entirely new skills from scratch.

The paper concludes that the best performance for solving new, tricky problems often happens early in the training process, and the second stage (RL) is mostly there to save us from the mistakes we made by training too long in the first stage.

Technical Summary: RL Fine-Tuning Heals OOD Forgetting in SFT

Problem Statement

The standard post-training pipeline for Large Language Models (LLMs) typically involves Supervised Fine-Tuning (SFT) followed by Reinforcement Learning Fine-Tuning (RLFT). A prevailing hypothesis in the field is that "SFT memorizes, RL generalizes," suggesting that SFT leads to overfitting on the training distribution while RL enhances Out-of-Distribution (OOD) generalization. However, the underlying dynamics of how OOD performance evolves during these two stages remain unclear. Specifically, it is not well understood whether RL creates new reasoning capabilities or merely recovers lost ones, nor is the mechanism behind OOD degradation during SFT fully characterized.

Methodology

The authors conduct a comprehensive checkpoint-wise analysis of the two-stage fine-tuning process using two open-source models: LLaMA-3.2-11B-Vision and Qwen-2.5-7B.

Experimental Setup:
- Tasks: The study utilizes controlled reasoning environments to explicitly define In-Distribution (ID) and OOD splits: GeneralPoints (arithmetic reasoning with modified card rules), Navigation (spatial reasoning with direction shifts), and Rank-Determinant Computation (linear algebra).
- Protocol: Models undergo full-parameter SFT, followed by RL (using PPO or GRPO). Crucially, the authors do not just compare the final SFT checkpoint to the final RL checkpoint. Instead, they apply RL starting from various intermediate SFT checkpoints to map the evolution of performance.
- Evaluation: Performance is tracked on both ID and OOD tasks across the entire training trajectory.
Mechanistic Analysis:
- SVD Analysis: The authors perform Singular Value Decomposition (SVD) on key parameter matrices (Self-Attention $W_Q, W_K, W_V$ and MLP layers). They conduct ablation studies where they restore singular values or singular vectors from specific checkpoints (e.g., swapping the singular vectors of a late SFT checkpoint with those of an early "peak" checkpoint) to isolate the impact of spectral changes on performance.
- Advantage Distribution Analysis: They analyze the distribution of advantage estimates during RL rollouts to understand the conditions under which RL successfully recovers OOD performance.

Key Findings

1. The "SFT Forgets, RL Recovers" Phenomenon

Contrary to the simple "SFT memorizes, RL generalizes" narrative, the authors observe a more nuanced dynamic:

Early SFT Peak: OOD reasoning performance often peaks very early in the SFT process (denoted as $SFT_{MaxOOD}$ ).
OOD Forgetting: As SFT continues beyond this peak, OOD performance degrades significantly, even while ID performance (training/test loss) continues to improve. This indicates that the model is over-specializing to the specific format or rules of the training data, effectively "forgetting" the general reasoning capability required for OOD tasks.
RL as a Restorer: RL does not typically surpass the best OOD performance achieved by early SFT. Instead, RL acts as a recovery mechanism, restoring the OOD capability lost during the later stages of SFT. This recovery is only effective within a bounded range of SFT checkpoints (e.g., where the ID accuracy is roughly between 40% and 80%).

2. Mechanism: Singular Vector Rotation

The spectral analysis reveals the root cause of these dynamics:

Stable Singular Values: The singular values of the parameter matrices remain largely stable throughout both SFT and RL stages. Fine-tuning does not significantly amplify or diminish the magnitude of these values.
Rotating Singular Vectors: The degradation and recovery of OOD performance correlate strongly with the rotation of singular vectors.
- During the "forgetting" phase of SFT, the singular vectors rotate away from the directions associated with OOD robustness.
- During RL, the singular vectors rotate back, partially restoring the original subspace alignment.
Ablation Evidence: Restoring the singular vectors of an early SFT checkpoint (when OOD was high) to a late SFT checkpoint (when OOD was low) recovers OOD performance, confirming that vector rotation is the primary driver of forgetting.

3. Advantage Distribution Constraints

The effectiveness of RL in recovering OOD performance depends on the distribution of advantage signals. RL is most effective when the advantage distribution is balanced (low skewness, high entropy, and low KL divergence from a normal distribution). If the SFT checkpoint is too advanced (policy entropy collapse) or too early (lack of task competence), the advantage signals become skewed or unstable, preventing effective recovery.

Significance and Claims

The paper claims to provide a more precise and mechanistic understanding of post-training dynamics:

Refining the Paradigm: It challenges the binary view of SFT vs. RL, proposing that SFT can forget and RL can recover. The best OOD performance is often found in early SFT checkpoints, not necessarily after RL.
Diagnostic Value: The study highlights that traditional overfitting detection methods (monitoring ID loss) fail to capture OOD forgetting, as ID loss may continue to decrease while OOD performance drops.
Algorithmic Implication: The findings suggest a potential direction for Protected SFT, where updates are orthogonalized against top-rank singular-vector directions associated with OOD capabilities to prevent their rotation and subsequent forgetting. Preliminary experiments with this approach show promise in preserving OOD accuracy while maintaining ID performance.
Generalizability: The "forgetting and recovery" pattern is observed across different models (LLaMA, Qwen), different reasoning tasks, and different RL algorithms (PPO, GRPO, DAPO), suggesting it is a fundamental property of the two-stage fine-tuning process.

In conclusion, the authors argue that RL is not a universal generator of new reasoning capabilities but rather a powerful tool for mitigating the OOD forgetting induced by prolonged SFT, provided it is initialized from a checkpoint with a balanced advantage distribution.