🤖 machine learning

SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning

The paper demonstrates that the standard SFT-then-RL pipeline actually outperforms recent mixed-policy methods, revealing that previous claims to the contrary were based on faulty baselines caused by specific bugs in DeepSpeed and OpenRLHF.

Original authors: Alexis Limozin, Eduard Durech, Torsten Hoefler, Imanol Schlag, Valentina Pyatkin

Published 2026-04-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Alexis Limozin, Eduard Durech, Torsten Hoefler, Imanol Schlag, Valentina Pyatkin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The "Broken Ruler" Mystery: Why AI Math Experts Weren't Actually Better Than They Looked

Imagine you are a judge at a high school math competition. You have two groups of students:

The "Traditionalists": They spend a month studying textbooks (Supervised Fine-Tuning) and then spend a week practicing hard problems with a coach who tells them "correct" or "incorrect" (Reinforcement Learning).
The "Innovators": They claim to have a revolutionary new method where they study textbooks and practice with the coach at the same time, blending the two styles to learn faster.

For the last year, the "Innovators" have been winning every trophy. Everyone in the AI world was cheering, "The new way is better! We should stop using the old textbook-then-coach method!"

But this paper just revealed a shocking truth: The Innovators weren't actually winning. The judges were using broken rulers to measure the Traditionalists.

The "Broken Rulers" (The Bugs)

The researchers discovered that the "Traditionalists" were being cheated by two massive technical glitches in the software used to train AI.

1. The "Selective Memory" Bug (The Optimizer Bug)

Imagine a student is doing 10 math problems to prepare for a test. They solve them one by one, adding up their progress in a notebook. But because of a glitch in their pen, the pen only writes down the very first problem and ignores the other nine. When the teacher checks the notebook, it looks like the student only learned one thing!

In AI terms, a bug in a popular tool called DeepSpeed caused the computer to "forget" most of the data it was processing during the textbook phase. It was only learning from a tiny fraction of what it was supposed to, making the AI look much "dumber" than it actually was.

2. The "Bad Averaging" Bug (The Loss Aggregation Bug)

Imagine a teacher grading a class. One student answers 2 questions, and another answers 100. Instead of looking at the total number of correct answers, the teacher just takes the average of each student's performance and treats them as equally important. This makes the student who did 100 questions look much less impressive than they actually are.

This bug in the OpenRLHF software caused the AI to weight its learning incorrectly, further dragging down its "score" during the textbook phase.

The Big Reveal: The Old Way is Actually Better

Once the researchers "fixed the rulers" (fixed the bugs), the results flipped completely.

When the "Traditionalists" (the standard SFT-then-RL method) were measured correctly, they didn't just catch up—they crushed the competition.

On one model (Qwen), the old way beat the "innovative" methods by a huge margin.
On another model (Llama), the old way was so much better it was almost like comparing a professional mathematician to a toddler.

Why does the old way work better?
The researchers explain it with a simple concept: Foundation first.

If you try to learn math by jumping straight into a high-stakes coaching session (Reinforcement Learning) without knowing the basics, you'll fail most of the time. The coach will just keep saying "Wrong!" and you'll learn nothing.

The "Traditional" way builds a solid foundation of knowledge first (the textbook phase). By the time the AI starts the coaching phase, it already knows enough to get many things right, which gives the coach much better "signals" to work with.

The Moral of the Story

This paper is a massive wake-up call for the AI industry. It tells us two things:

Don't assume a "new" method is better just because the old one looks bad. Always check if the "old" method was being measured fairly.
Sometimes, the simplest way is the best way. Instead of trying to invent complex, "blended" ways of learning, we should focus on making sure the fundamental stages—learning the facts, then practicing the skills—are done perfectly.

Technical Summary: SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning

1. Problem Statement

In the training of Large Language Models (LLMs) for complex reasoning (e.g., mathematics), the current industry standard is a two-stage pipeline: Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL).

Recently, a wave of research has proposed "mixed-policy" methods. These methods attempt to interleave or blend SFT and RL signals during a single training stage to solve the "sparse reward" problem (where RL struggles because the model rarely finds the correct answer) and the "lack of exploration" problem (where SFT alone cannot develop autonomous reasoning). However, these new methods consistently report state-of-the-art (SOTA) results by claiming significant improvements over the standard SFT-then-RL baseline.

2. Methodology & Discovery of Bugs

The authors of this paper hypothesize that the perceived superiority of mixed-policy methods is an illusion caused by faulty baselines. Through rigorous cross-framework validation, they identify two critical, silent bugs in widely used open-source training frameworks that systematically deflate SFT performance:

The CPU-Offloaded Optimizer Bug (DeepSpeed): In configurations using DeepSpeed ZeRO Stage 1 or 2 with CPU offloading (to save GPU memory), a bug in the gradient accumulation routine causes the optimizer to only "see" the gradients from the first micro-batch. The gradients from subsequent micro-batches are accumulated on the GPU but never copied to the CPU-side optimizer. This effectively renders the majority of the training data useless for gradient updates. This bug affects major frameworks like TRL, OpenRLHF, and Llama-Factory.
The Loss Aggregation Bug (OpenRLHF/Llama-Factory/verl): A common error in gradient accumulation where the code computes a "mean of per-mini-batch means" rather than a true "per-token mean." Because different mini-batches contain different numbers of active response tokens, this incorrectly weights mini-batches, leading to suboptimal optimization and training instability.

3. Key Contributions

Bug Identification and Fixes: The authors identified and provided fixes for the two bugs mentioned above. They submitted pull requests to DeepSpeed and OpenRLHF, which were subsequently merged.
Baseline Correction: They re-implemented and corrected the SFT baselines using the verl framework and a patched version of OpenRLHF.
Comparative Analysis: They conducted an extensive evaluation across two model families (Qwen2.5-Math-7B and Llama-3.1-8B) and multiple mathematical benchmarks (AIME, AMC, MATH-500, etc.) to compare the corrected SFT-then-RL pipeline against published mixed-policy methods (LUFFY, ReLIFT, SRFT, etc.).

4. Results

The results demonstrate that once the bugs are fixed, the standard SFT-then-RL pipeline is significantly superior to mixed-policy methods:

Performance Gains: On Qwen2.5-Math-7B, the corrected SFT-then-RL pipeline outperforms the best published mixed-policy method (SRFT) by +3.8 points on in-distribution math tasks. On Llama-3.1-8B, the gap is even more massive: the corrected pipeline outperforms the best competitor by +22.2 points.
Efficiency (FLOPs): The authors found that a "truncated" SFT-then-RL pipeline (using only 50 RL steps instead of 500) still outperforms all mixed-policy methods while using significantly fewer FLOPs.
Training Dynamics: The paper shows that SFT provides a "dense" reward signal for the subsequent RL stage. In contrast, mixed-policy methods struggle because they attempt to learn knowledge and refine reasoning simultaneously, often starting from a near-zero reward state.

5. Significance

This paper has profound implications for the LLM research community:

Paradigm Validation: It restores confidence in the sequential SFT $\rightarrow$ RL paradigm, suggesting that "separating concerns" (knowledge acquisition via SFT and strategy refinement via RL) is more effective than attempting to do both at once.
Scientific Rigor: It serves as a cautionary tale regarding "silent bugs" in deep learning frameworks. Because these bugs do not cause crashes but merely degrade performance, they can lead to false scientific conclusions and the promotion of suboptimal methodologies.
Call for Cross-Validation: The authors emphasize the necessity of validating results across independent, diverse frameworks (e.g., comparing DeepSpeed against PyTorch FSDP) to ensure that reported "innovations" are not simply the result of fixing a broken baseline.