💬 NLP

Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling

This paper introduces Prefix-RFT, a hybrid fine-tuning method that synergizes supervised fine-tuning and reinforcement fine-tuning through prefix sampling to overcome the limitations of each individual approach, demonstrating superior performance and robustness on mathematical reasoning tasks.

Original authors: Zeyu Huang, Tianhao Cheng, Zihan Qiu, Zili Wang, Yinghui Xu, Edoardo M. Ponti, Ivan Titov

Published 2026-05-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zeyu Huang, Tianhao Cheng, Zihan Qiu, Zili Wang, Yinghui Xu, Edoardo M. Ponti, Ivan Titov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart but inexperienced student how to solve complex math puzzles. You have two main ways to teach them, but both have a major flaw on their own.

The Two Flawed Teaching Styles

The "Copycat" Method (Supervised Fine-Tuning or SFT):
You sit the student down with a textbook of perfect solutions. You say, "Read this, memorize it, and write it down exactly as it is."
- The Good: The student learns the correct format and facts very quickly.
- The Bad: The student becomes a robot. If they see a slightly different puzzle, they freeze because they only know how to copy, not how to think. They mimic the book but don't truly understand the logic.
The "Trial-and-Error" Method (Reinforcement Fine-Tuning or RFT):
You throw the student into the deep end. You say, "Go solve these puzzles. If you get the right answer, you get a gold star. If you get it wrong, you get nothing."
- The Good: The student learns to think creatively and find new solutions. They become very good at solving problems they've never seen before.
- The Bad: Without a guide, the student might develop weird habits. They might start speaking in a mix of languages or take bizarre, inefficient paths just to get a gold star. Also, if they start with a bad habit, it's very hard to break.

The New Solution: The "Prefix" Coach (Prefix-RFT)

The authors of this paper propose a new method called Prefix-RFT. Think of it as a hybrid coach who uses a "Start-Hint" strategy.

Here is how it works, using a simple analogy:

Imagine the student is trying to solve a maze.

The Old Way (SFT): You hand them a map of the entire maze and tell them to memorize the path.
The Old Way (RFT): You blindfold them and tell them to walk until they find the exit.
The New Way (Prefix-RFT): You give them a map, but only for the first 20% of the maze. You say, "Start exactly like this map shows. Once you reach this point, put the map away and figure out the rest of the maze on your own."

Why is this brilliant?

It gives a safe start: By forcing the student to follow the expert's path for the beginning, you prevent them from wandering off into a dead end immediately (fixing the RFT problem of "weird habits").
It forces independent thinking: Because the student must figure out the rest of the maze themselves, they don't just become a copycat. They have to use their brain to finish the job (fixing the SFT problem of "lack of generalization").
The "Gold Star" Logic: If the student follows the expert's start and finds a great solution for the rest, they get a huge reward. This teaches the model that the expert's start was a good idea, but the student's own finish was also valuable.

The "Entropy" Filter (The Safety Valve)

The paper mentions a tricky technical detail called "Entropy-based Clipping." Here is the simple version:

Sometimes, the expert's map is so different from what the student knows that if the student tries to copy it, they might get confused and break their brain (mathematically, the "gradients" get too huge).

To fix this, the coach only lets the student copy the parts of the map where they are unsure.

If the student already knows the first step perfectly, the coach says, "Skip that, you know it."
If the student is confused about a step, the coach says, "Look at the expert's move here, this is where you need to learn."

This ensures the student learns from the expert only where they actually need help, without being overwhelmed.

What the Results Show

The authors tested this on math problems (like the AIME competition).

The Copycat (SFT) was okay but couldn't handle hard new problems.
The Trial-and-Error (RFT) was good but sometimes got stuck or developed bad habits.
The Prefix Coach (Prefix-RFT) beat both of them. It learned the expert's patterns and kept the ability to explore new solutions.

In fact, when they gave the model a huge number of tries (like 2,048 attempts) to solve a single hard problem, the Prefix Coach was the only method that significantly improved the chances of finding the solution. It proved that this method doesn't just make the model better at copying; it actually raises the ceiling of what the model is capable of achieving.

In a Nutshell
Prefix-RFT is like giving a student a "training wheel" for the first part of a journey, then taking them off so they can ride the rest of the way on their own. It combines the safety of a guide with the freedom of exploration, resulting in a learner who is both knowledgeable and creative.

Technical Summary: Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling

1. Problem Statement

Large Language Model (LLM) post-training is currently dominated by two distinct paradigms: Supervised Fine-Tuning (SFT) and Reinforcement Fine-Tuning (RFT). Each approach presents specific limitations:

SFT excels at mimicking demonstration data but functions as behavioral cloning, often leading to problematic generalization and robustness issues. It provides dense supervision but may align poorly with downstream task performance.
RFT (e.g., PPO, GRPO) enhances performance through exploration and goal-oriented optimization but suffers from sparse reward signals, sensitivity to the initial policy, and the risk of learning unexpected behaviors (e.g., language mixing). Recent literature questions whether RFT can truly raise a model's intrinsic capability ceiling or merely refines existing pre-training knowledge.

The core research question addressed is how to formally integrate the process supervision of SFT with the goal-oriented optimization of RFT to leverage their complementary strengths: SFT for expanding knowledge boundaries and RFT for steering toward robust problem-solving.

2. Methodology: Prefix-RFT

The authors propose Prefix-RFT, a hybrid post-training approach that unifies SFT and RFT within a single optimization framework.

Unified View

The paper establishes that SFT and RFT share a consistent optimization structure. Both update the log-probability of sequences, differing primarily in the source of the sequence (expert vs. on-policy) and the weighting mechanism (fixed vs. advantage-weighted).

Core Mechanism

Prefix-RFT incorporates offline demonstration datasets into the RFT training loop via Prefix Sampling:

Hybrid Trajectory Construction: For a given prompt $x$ and demonstration $y^*$ , the policy $\pi_{\theta}$ generates $N-1$ standard on-policy rollouts. For the $N$ -th sequence, a prefix $y^*_{<L}$ is sampled from the demonstration. The policy then generates a continuation $y_{\ge L}$ , creating a hybrid trajectory $y^{(N)} = (y^*_{<L}, y_{\ge L})$ .
Unified Gradient Update: The hybrid trajectory is treated alongside standard rollouts. The gradient update combines learning from exploration (on-policy tokens) and imitation (prefix tokens).
- Prefix Tokens ( $t < L$ ): Treated as "imitation" but weighted by the advantage $\hat{A}_t$ estimated from the entire hybrid trajectory. If the hybrid trajectory yields a high reward, the prefix is reinforced; otherwise, its influence is dampened.
- Continuation Tokens ( $t \ge L$ ): Treated as standard "exploration" tokens.
Entropy-Based Clipping: To prevent off-policy demonstration tokens from dominating the gradient (due to distributional shifts causing large log-likelihood gradients), the method updates only the top- $k\%$ (e.g., 20%) of prefix tokens with the highest entropy. Low-entropy tokens are either already matched by the model or represent confident mismatches that could cause sharp, destabilizing overwrites.
Cosine Decay Scheduler: To mitigate positional bias (where models rarely see the end of sequences) and implement curriculum learning, the prefix length $L$ is controlled by a scheduler. The prefix length starts long (high guidance) and decays to short (more autonomy) throughout training, mimicking a transition from SFT to RFT.

3. Key Contributions

Unified Framework: The paper provides a theoretical unification of SFT and RFT, demonstrating that they can be blended via a hybrid objective rather than treated as consecutive stages.
Prefix-RFT Algorithm: Introduces a simple yet effective method to inject offline demonstrations into RFT without rigidly constraining the model, allowing it to explore superior continuations while guided by expert prefixes.
Dynamic Weighting & Clipping: Proposes using trajectory-level advantages to dynamically weight prefix updates and employs entropy-based clipping to stabilize training against off-policy distribution shifts.
Empirical Validation: Demonstrates that Prefix-RFT outperforms standalone SFT, RFT, and sequential SFT-then-RFT baselines, as well as recent parallel hybrid methods (e.g., UFT, LUFFY, ReLIFT).

4. Experimental Results

Experiments were conducted primarily on mathematical reasoning tasks (AIME, AMC, MATH-500, etc.) using models like Qwen2.5-Math (1.5B, 7B) and LLaMA-3.1-8B.

Performance: Prefix-RFT consistently outperformed all baselines across six math reasoning benchmarks and three general domain reasoning tasks. For instance, on Qwen2.5-Math-7B, it achieved a Pass@1 score of 31.8 on AIME 2024 and 26.4 on AIME 2025, surpassing the best baseline (LUFFY) and the SFT+RFT sequential approach.
Reasoning Boundaries (Pass@k): Crucially, Prefix-RFT was the only method that significantly raised the model's performance upper bound. On AIME 2025, it achieved a Pass@2048 of 76.7%, compared to 70.0% for the base model and RFT, indicating it expands the model's reasoning capabilities rather than just increasing the likelihood of known solutions.
Robustness: The method showed strong robustness across different model scales and was effective even with reduced demonstration quantities (down to 1% of data) and varying quality of demonstration generators (from 1.5B to 32B models).
Generalization: Preliminary results on code generation (MBPP+, HumanEval+) confirmed the method's applicability beyond mathematical reasoning.

5. Analysis and Significance

The paper's analysis highlights the complementary nature of SFT and RFT:

Bridging the Gap: Prefix-RFT effectively synthesizes the distinct training paradigms. It allows the model to learn from demonstrations (SFT-like) while maintaining the goal-oriented objective of RFT.
Dynamic Transition: The method induces a dynamic transition where the model relies heavily on prefixes early in training (high advantage, long prefixes) and gradually shifts toward autonomous exploration as its capabilities improve (diminishing advantage, shorter prefixes).
Targeted Learning: The model learns more intensively from demonstrations on challenging problems (where it struggles) and relies more on self-exploration for easier problems, validating the adaptive nature of the approach.

Significance:
The authors claim that Prefix-RFT offers a "sweet spot" between the stability of SFT and the exploratory power of RFT. By formally integrating these paradigms, the method addresses the limitations of behavioral cloning (SFT) and the instability of pure reinforcement learning (RFT). The work suggests that treating SFT and RFT as distinct, consecutive stages is suboptimal, and that an integrated approach can effectively harmonize them to push the boundaries of LLM reasoning capabilities.

Limitations:
The authors note that their experiments focus on verifiable reasoning settings with reliable outcome rewards. While code generation results suggest transferability, broader open-ended generation and noisy-reward settings remain future work. Additionally, the method relies on external demonstrations, and systematic selection of multiple candidate demonstrations is left for future study.