🤖 machine learning

RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training

This paper challenges the standard practice of applying reinforcement learning (RL) only after supervised fine-tuning (SFT) by demonstrating that integrating RL earlier in pre-training, optimizing data composition, and merging RL with SFT objectives can effectively enhance reasoning capabilities and expand model distributions while preserving general abilities.

Original authors: Rachit Bansal, Clara Mohri, Tian Qin, David Alvarez-Melis, Sham Kakade

Published 2026-06-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rachit Bansal, Clara Mohri, Tian Qin, David Alvarez-Melis, Sham Kakade

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are training a brilliant but raw student (the AI model) to solve math problems. For years, the standard recipe has been three steps:

Pre-training: The student reads millions of books to learn general language and facts.
SFT (Supervised Fine-Tuning): A teacher gives the student a textbook with problems and their exact correct answers, forcing the student to memorize the solution path.
RL (Reinforcement Learning): The student is put in a game where they get points only for getting the final answer right, encouraging them to figure out new ways to solve problems on their own.

This paper asks: Do we really have to wait until the student has read all the books and memorized the textbook before we let them play the game?

Here is what the researchers found, explained simply:

1. You Can Start the Game Early

The standard rule says, "Wait until the student is fully trained before giving them the game." The researchers tried giving the game (RL) to the student while they were still reading their first few chapters (early in pre-training).

The Result: It worked surprisingly well! Even when the student had only read a tiny fraction of the books, playing the game helped them solve math problems much better than just reading more books. In fact, playing the game early was often just as good as waiting until the end and doing the whole "Read → Memorize → Play" routine.

2. The "Game" is Better When You Don't Have a Textbook

Usually, the "Memorize" step (SFT) requires a textbook with the correct answers. But what if you don't have enough textbooks?

The Finding: If you only have one or two examples of how to solve a problem, the "Memorize" step fails. But the "Game" (RL) thrives. The student learns to explore and find solutions on their own without needing a perfect answer key. The game is a much stronger teacher when you are short on examples.

3. The Secret Ingredient is the Type of Books, Not the Student's Size

People often think, "If the student is bigger (more powerful), they will learn better." The researchers tested this by making the student bigger.

The Twist: Making the student bigger didn't help them learn the game faster. However, giving them more math books specifically during their early reading phase did.
The Lesson: If you want the student to be good at math, feed them math stories early on. The content of what they read matters more than how big their brain is.

4. The "Sharpening" Myth vs. "Expansion"

There is a recent debate: Does the "Game" (RL) just make the student's existing answers sharper and more confident, or does it actually teach them new ways of thinking?

The Old View: Many thought RL just "sharpened" the student, making them confident but not necessarily smarter.
The New Discovery: The researchers found that the "sharpening" effect actually happens because of the "Memorize" step (SFT). When you force a student to memorize a textbook first, they stop exploring.
The Real Magic: If you let the student play the game directly without memorizing the textbook first, they actually expand their thinking. They try many different paths and discover new solutions they never knew before. The "Memorize" step is what actually limits their creativity, not the game itself.

5. The "Super-Student" Recipe

The researchers combined the best of both worlds. Instead of doing "Memorize" then "Play," they had the student do both at the same time.

How it works: Imagine the student is solving a problem. One part of their brain is checking the textbook (SFT), and another part is experimenting with new ideas (RL). They average the advice from both parts to update their brain.
The Result: This "Super-Student" (Parallel Averaging) became the best at solving problems. They got the right answers more often than anyone else, and unlike the students who just memorized the textbook, they didn't forget how to do other things (like general conversation).

Summary of the Big Takeaways

Don't wait: You can use Reinforcement Learning (the game) very early in a model's life, even before it has finished its "reading" phase.
Data matters more than size: Feeding the model specific types of data (like math) during pre-training is more important than just making the model bigger.
RL isn't the villain: RL doesn't ruin a model's general skills or just "sharpen" it. Those negative effects come from the "Memorize" (SFT) step. RL actually helps models explore and find new solutions.
Do it together: The best results come from mixing the "Memorize" and "Play" steps simultaneously, rather than doing them one after the other.

In short, the paper suggests we should stop treating Reinforcement Learning as a final polish at the very end. Instead, we should weave it into the training process much earlier and mix it with other methods to get smarter, more capable models.

Technical Summary: RL Excursions during Pre-Training

Problem Statement

The standard training pipeline for Large Language Models (LLMs) typically restricts Reinforcement Learning (RL) to the post-training phase, applied only after extensive next-token prediction (NTP) pre-training and Supervised Fine-Tuning (SFT). This sequential approach raises two fundamental questions:

Timing: Is it fundamentally necessary to wait until the end of pre-training to apply RL, or can RL be effective earlier?
Mechanism: Does RL inherently "sharpen" a model's existing output distribution (limiting exploration), or does it expand capabilities? Recent literature suggests RL primarily sharpens distributions, but it is unclear if this is an artifact of the standard SFT→RL pipeline rather than the RL objective itself.

Furthermore, there is a debate regarding whether RL erodes general capabilities inherited from pre-training.

Methodology

The authors conduct a large-scale, controlled study to isolate the effects of RL from the standard training regime.

Model & Data: They pre-train a 1B parameter model (based on OLMo2 architecture) from scratch on a high-quality, reasoning-heavy corpus (DOLMino mix, 50B tokens). They also experiment with a 4B model and a 1B model trained on 60B tokens (with additional math data).
Checkpoints: Instead of waiting for the final model, they save checkpoints throughout the pre-training process (from 4B to 50B tokens).
Training Pipelines: For each checkpoint $M_t$ $M_{t}$ , they apply four distinct post-training strategies:
1. Direct RL ( $M^R_t$ ): Applying RL (specifically GRPO with verifiable rewards) directly to the base checkpoint without prior SFT.
2. SFT ( $M^{SFT}_t$ ): Supervised Fine-Tuning using a single ground-truth solution per problem.
3. SFT-Gold ( $M^{SFT-Gold}_t$ ): SFT using all available ground-truth solutions per problem (a setting often impractical in reality).
4. Standard Pipeline ( $M^{SFT \to RL}_t$ ): SFT followed by RL, representing the current industry standard.
5. Parallel Averaging ( $M^{Parallel}_t$ ): A proposed method where gradients from RL and SFT are computed in parallel from the same parameter snapshot and averaged to update weights.
Evaluation: Performance is measured on GSM8K (grade-school level) and MATH (competition level) using pass@k metrics ( $k=1, 8, 32$ ). General capabilities are evaluated on six non-math benchmarks (e.g., LAMBADA, HellaSwag).

Key Contributions & Results

1. RL is Effective Early in Pre-Training

Contrary to the assumption that RL requires a highly capable base model, the authors find that RL is effective surprisingly early.

Early Gains: Direct RL improves performance on GSM8K starting from as few as 4B pre-training tokens, well before the Chinchilla-optimal token count.
Competitiveness: By 10B tokens, Direct RL matches the performance of the standard SFT→RL pipeline on pass@1 and pass@32, despite never seeing ground-truth reasoning traces.
Data Scarcity: When ground-truth demonstrations are scarce (single solution per prompt), RL significantly outperforms SFT. SFT only surpasses RL when provided with multiple high-quality solutions (SFT-Gold), a condition rarely met in practice.

2. Pre-Training Data Composition > Model Scale

For harder reasoning tasks (MATH dataset), the effectiveness of RL depends more on the composition of pre-training data than on model scale.

Scaling Data: Adding targeted math-heavy data to the pre-training mix (increasing tokens from 50B to 60B) significantly boosts the gains achievable by Direct RL.
Scaling Model: Increasing model size from 1B to 4B improves the base model's absolute performance but does not substantially improve the relative gain provided by RL.
Diagnostic: The base model's pass@k accuracy on the test set serves as a lightweight predictor for whether RL training will yield downstream gains.

3. RL Expands vs. Sharpens Distributions

The paper challenges the claim that RL inherently sharpens distributions.

Expansion: When applied directly to base checkpoints, RL expands the model's distribution, improving both pass@1 and pass@32. The model discovers new reasoning paths through on-policy learning.
Sharpening Artifact: The "sharpening" effect (improved pass@1 but degraded pass@32) observed in recent work arises only when RL follows SFT. The authors hypothesize that SFT constrains exploration by exposing the model to ground-truth solutions, causing subsequent RL to merely refine existing paths rather than discover new ones.

4. Preservation of General Capabilities

SFT Degradation: SFT consistently degrades general (non-math) capabilities by 4–8 percentage points across benchmarks.
RL Preservation: Direct RL leaves general capabilities essentially unchanged.
Parallel Averaging: The proposed parallel averaging method ( $M^{Parallel}_t$ ) combines the strengths of both: it achieves the strongest pass@32 scores (surpassing the standard pipeline) while preserving general capabilities, avoiding the regression seen in SFT-based regimes.

Significance

The paper argues that many assumptions about RL (e.g., that it requires a mature base model, that it inherently sharpens distributions, or that it erodes general capabilities) are artifacts of the current SFT→RL training regime rather than inherent properties of the RL objective.

The authors conclude that:

Feasibility: Introducing RL earlier in the training pipeline is feasible and often preferable.
Data Levers: Targeted pre-training data composition is a stronger lever for RL effectiveness than model scaling.
Objective Synergy: RL and SFT objectives are complementary. The parallel averaging approach demonstrates that combining them can yield superior reasoning capabilities (pass@32) without sacrificing general intelligence.

The work suggests a paradigm shift where RL is treated as a first-class training objective integrated throughout the pre-training and fine-tuning process, rather than a final post-training step.