💻 computer science

SFT Overtraining Predicts Rank Inversion via Entropy Collapse Under RLVR

This paper demonstrates that excessive Supervised Fine-Tuning (SFT) can compress the rollout distribution, causing entropy collapse that leads to rank inversion and degraded performance in subsequent Group Relative Policy Optimization (GRPO) training, a failure mode that can be predicted and mitigated by monitoring pre-RL entropy and early GRPO dynamics.

Original authors: Siddharth Aphale, Kelly Liu

Published 2026-06-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Siddharth Aphale, Kelly Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Too Perfect" Trap

Imagine you are training a robot to write code. The standard recipe is two steps:

Supervised Fine-Tuning (SFT): You show the robot thousands of examples of correct code so it learns the patterns.
Reinforcement Learning (RL): You let the robot try to solve new problems on its own. If it gets it right, it gets a treat (a reward); if it fails, it gets nothing.

The Problem: The paper found that if you train the robot too much in Step 1 (Step 1 Overtraining), it actually becomes worse at Step 2.

Usually, we think, "The more examples the robot memorizes in Step 1, the better it will be." The authors discovered the opposite: The robot that memorizes the most in Step 1 actually performs the worst in Step 2. In fact, the robot that memorizes the least ends up being the champion.

The Metaphor: The Stiff Dancer vs. The Flexible Dancer

To understand why this happens, imagine two dancers preparing for a competition where they must improvise a routine based on random music.

The "Overtrained" Dancer (Deep SFT): This dancer practiced the same 10 perfect routines thousands of times. They are incredibly precise. But when the music changes, they freeze. They only know those 10 moves. They have no idea how to move in any other way.
The "Just-Right" Dancer (Shallow SFT): This dancer practiced the basics but kept their body loose and flexible. They know the rules, but they can still wiggle, twist, and try new things when the music changes.

The Competition (RL):
The competition requires the dancers to try many different moves at once to see which one works best.

The Overtrained Dancer tries to do the exact same perfect move every time. Since they all look the same, the judges (the algorithm) can't tell which move is better. The dancer gets stuck and can't learn anything new.
The Flexible Dancer tries a wide variety of moves. The judges can see, "Ah, this move worked, that one didn't!" and the dancer learns quickly.

The Technical "Why": Entropy Collapse

The paper uses a fancy word for "variety" or "flexibility": Entropy.

Entropy Collapse: When the robot is overtrained, its "mind" becomes too narrow. It stops exploring different possibilities and just repeats the same few answers. In the paper's terms, its entropy collapses.
The Signal Dies: The learning algorithm (called GRPO) works by comparing different answers. If the robot gives the same answer 8 times in a row, the algorithm gets confused. It can't calculate a "gradient" (a direction to improve) because there is no difference to compare. It's like trying to steer a car when the steering wheel is locked straight ahead.
The Rank Inversion: This leads to a weird result called Rank Inversion.
- Before the second training phase, the overtrained robot looks like a genius (it gets the right answer 100% of the time on the practice test).
- After the second training phase, it becomes a disaster.
- Meanwhile, the "less perfect" robot, which looked slightly worse at first, becomes the superstar.

The Two Models: A Tale of Two Cities

The authors tested this on two different robot brains (models):

Qwen (The Victim): This model fell into the trap. The more they trained it in Step 1, the more it stiffened up. The "best" looking checkpoint (highest score) turned out to be the worst starting point for the next phase.
DeepSeek (The Survivor): This model was naturally more flexible. Even after heavy training, it didn't get stiff. It stayed in the "safe zone" where it could still learn. This proved that the problem wasn't the second training phase itself, but specifically how the first phase ruined the Qwen model's flexibility.

The Solution: A Two-Step Checkup

The authors didn't just find the problem; they built a diagnostic tool to spot it before wasting time and money.

Step 1: The "Stretch" Test (Pre-RL Entropy): Before starting the second training phase, they check how "loose" the robot's mind is. If the robot is too rigid (low entropy), they flag it as high risk.
Step 2: The "Early Warning" Monitor: If they start the second phase, they watch the first few minutes. If the robot stops trying new things and starts repeating itself immediately, they stop the training early to save resources.

What Didn't Work?

The authors tried to fix the broken robot with standard tricks, like:

Adding a penalty: "Don't deviate too much from your original training." (This made it worse).
Smoothing the labels: "Be a little less certain about your answers." (This made the robot look confident again, but it still failed the competition).

The Conclusion: You can't fix a stiff dancer by telling them to "try harder" during the competition. You have to stop training them so rigidly in the first place. The paper proves that variety (entropy) is more important than perfection when you want a model to keep learning.

Technical Summary: SFT Overtraining Predicts Rank Inversion via Entropy Collapse Under RLVR

Problem Statement
The standard heuristic for Reinforcement Learning with Verifiable Rewards (RLVR) in code generation involves selecting the Supervised Fine-Tuning (SFT) checkpoint with the highest pass@1 score to initialize the RL phase (e.g., GRPO). This paper demonstrates that this heuristic fails when SFT overtraining compresses the model's rollout distribution. Specifically, prolonged SFT leads to "entropy collapse," where the model's output diversity diminishes to the point that the RL algorithm cannot generate a meaningful gradient signal. In this regime, selecting the highest pass@1 checkpoint systematically identifies the worst-performing RL initializer, resulting in a "rank inversion" where shallower SFT checkpoints outperform deeper ones after RL training.

Methodology
The study isolates SFT duration as the sole independent variable across two model families: Qwen2.5-Coder-3B-Base and DeepSeek-Coder-6.7B-Base.

Experimental Setup: Both models were fine-tuned on 5,000 medium-difficulty KodCode-V1 problems. Checkpoints were selected at varying SFT depths (1.0 to 5.8 epochs for Qwen; 1.0 to 9.6 epochs for DeepSeek).
RL Training: All checkpoints underwent identical GRPO training (400 steps, group size $g=8$ , binary correctness reward) on a held-out dataset.
Metrics: Performance was measured via pass@1 and pass@10 (using an unbiased estimator) on a frozen 40-problem HumanEval+ subset. Crucially, the authors tracked pre-RL entropy (mean next-token entropy at $T=1.0$ ) and in-training entropy to diagnose distribution collapse.
Interventions: To test if the failure was a hyperparameter artifact, the authors applied a KL penalty to the reference policy during GRPO and label smoothing during SFT to artificially restore entropy.

Key Contributions

Theoretical Threshold for Gradient Collapse: The authors derive that for binary rewards, the expected within-group advantage variance is $E[\sigma^2_G] = p(1-p)(g-1)/g$ . They identify a critical threshold $p^*(g)$ where, if the probability of a correct rollout ( $p$ ) drops below this value, the group relative signal structurally collapses (most groups receive identical rewards), causing the gradient to vanish.
Rank Inversion Case Study: On Qwen2.5-Coder-3B, the paper documents a clear rank inversion: as SFT depth increases, pre-RL pass@1 rises, but peak post-RL pass@10 falls monotonically (from 0.806 to 0.481). The highest pass@1 checkpoint (5.8 epochs) performs worst under RL.
Contrastive Validation: On DeepSeek-Coder-6.7B, where pre-RL pass@1 remains significantly above the theoretical collapse threshold $p^*(8)$ , no rank inversion occurs. Instead, ranks compress, and all checkpoints improve, validating the threshold hypothesis.
Two-Stage Diagnostic: The authors propose a diagnostic protocol to identify high-risk checkpoints before or early in RL:
- Stage 1 (Pre-RL): Flag checkpoints where mean next-token entropy falls below a threshold ( $\tau_H = 0.18$ nats).
- Stage 2 (Early RL): Monitor the relative drop in worst-token entropy between GRPO steps 10 and 150. A drop exceeding 50% indicates collapse.
Failure Localization: The study shows that simple interventions (KL penalties or label smoothing) fail to rescue collapsed checkpoints, suggesting the failure is rooted in the SFT stage's depletion of plasticity and entropy, rather than a tunable GRPO hyperparameter issue.

Results

Qwen2.5-Coder-3B: A strong negative correlation ( $\rho = -0.75$ ) exists between pre-RL pass@1 and peak post-RL pass@10. The 5.8-epoch checkpoint, despite having the highest pre-RL pass@1 (0.187), collapsed to a peak pass@10 of 0.481, while the 1.0-epoch checkpoint achieved 0.806. The entropy of the 5.8-epoch model dropped by 78% during training, compared to 37% for the 1.0-epoch model.
DeepSeek-Coder-6.7B: All checkpoints maintained pre-RL pass@1 well above the collapse threshold ( $p \geq 4.2 \times p^*(8)$ ). Consequently, all arms improved under GRPO, and the pre-RL ranking was perfectly preserved post-RL ( $\rho = +1.00$ ).
Interventions: Adding a KL penalty ( $\beta=0.01$ ) reduced performance for both shallow and deep checkpoints but did not reverse the rank inversion. Label smoothing on the 5.8-epoch checkpoint increased pre-RL pass@1 to 0.600 but resulted in the lowest post-RL pass@10 (0.214), indicating that artificially flattening the distribution without restoring the underlying solution diversity is insufficient.

Significance and Claims
The paper claims that the standard practice of selecting the highest pass@1 SFT checkpoint for RL is misleading in regimes where SFT overtraining leads to entropy collapse. The authors argue that pass@1 measures capability at the wrong temperature (greedy, $T=0$ ) and fails to capture the diversity required for RL at $T=1.0$ .

The significance lies in identifying entropy collapse as the mechanistic link between SFT overtraining and RL failure. The paper posits that the failure is not a trivial artifact of GRPO hyperparameters but a fundamental issue of plasticity loss during SFT. The proposed two-stage diagnostic offers a practical, model-agnostic method (based on ordering and risk signals rather than universal thresholds) to filter out checkpoints that have already exhausted their learning signal before RL compute is expended. The authors conclude that preserving entropy during SFT is a prerequisite for successful RLVR, and that once entropy collapses, standard RL variants cannot recover the policy.