🤖 AI

Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models

This paper presents a unified framework for Large Language Model post-training by comprehensively analyzing the distinct yet interconnected roles of Supervised Fine-Tuning and Reinforcement Learning, exploring their hybrid integration, and identifying key trends and best practices for effective application based on recent empirical evidence.

Original authors: Haitao Jiang, Wenbo Zhang, Jiarui Yao, Hengrui Cai, Sheng Wang, Rui Song

Published 2026-03-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Haitao Jiang, Wenbo Zhang, Jiarui Yao, Hengrui Cai, Sheng Wang, Rui Song

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, well-read student named LLM. This student has read almost every book in the library (the pre-training phase). They know a lot of facts, can write poetry, and understand grammar. But, if you ask them to solve a specific, tricky math problem or write code for a specific robot, they might get it wrong, hallucinate (make things up), or act too casually.

To turn this "general student" into a "specialist expert," you need to give them extra training. This paper is a guidebook on the two main ways to do that training: SFT and RL.

Here is the breakdown using simple analogies:

1. The Two Training Methods

SFT (Supervised Fine-Tuning): The "Copy the Master" Method

The Analogy: Imagine you want to teach your student to play chess. You give them a stack of 1,000 games played by a Grandmaster. You say, "Look at move A, then move B. Just copy exactly what the master did."
How it works: The student learns by imitating perfect examples. It's fast, stable, and great for learning the "rules" and basic patterns.
The Downside: If the student encounters a chess board situation that wasn't in the 1,000 examples, they might freeze or make a weird move because they are just memorizing, not truly understanding why a move is good. They might also get stuck in a loop of repeating the master's mistakes if the examples weren't perfect.

RL (Reinforcement Learning): The "Trial, Error, and Reward" Method

The Analogy: Now, put the student in a real chess tournament. They play a game. If they win, they get a gold star (Reward). If they lose, they get a red card (Penalty). They don't have a script; they have to figure out the best moves by trying things, failing, and learning from the feedback.
How it works: The student explores different strategies. If a move leads to a win, they remember it. If it leads to a loss, they avoid it. This helps them generalize and handle new, weird situations they've never seen before.
The Downside: It's chaotic and expensive. The student might try thousands of terrible moves before finding a good one. They might also learn to "cheat" (hack the reward) to get gold stars without actually playing well.

2. The Big Discovery: They Are Cousins, Not Enemies

For a long time, researchers thought SFT and RL were totally different tools. This paper argues they are actually two sides of the same coin.

The Insight: You can actually view "copying the master" (SFT) as just a special, simple version of "getting rewards" (RL). In SFT, the "reward" is simply: "Did you copy the answer exactly? Yes = Good, No = Bad."
The Takeaway: Because they are so similar, the tricks used to make one work better often work for the other.

3. The Winning Strategy: The Hybrid Approach

The paper observes a massive trend from 2023 to 2025: Stop choosing one; use both.

Think of it like training an athlete:

Phase 1 (SFT): First, you have them watch hours of Olympic footage and copy the perfect form. This gets them on the right track quickly.
Phase 2 (RL): Then, you put them in a real competition. They use what they learned from the videos but adapt to the pressure and the specific moves of their opponent, getting feedback on what actually works.

Why combine them?

SFT gives them a solid foundation so they don't start from zero.
RL helps them refine that foundation, fix their mistakes, and become creative problem-solvers.

4. Where is this happening?

The paper looks at four main "sports" where this training is happening:

General Q&A: Like a trivia champion. (SFT teaches facts; RL teaches how to say "I don't know" instead of making things up).
Math: Like a math whiz. (SFT teaches the formulas; RL teaches the step-by-step logic to solve a problem they've never seen).
Agents (Robots): Like a personal assistant. (SFT teaches how to open a door; RL teaches how to navigate a room full of obstacles to get a cup of coffee).
Coding: Like a software engineer. (SFT teaches syntax; RL teaches how to debug code that breaks).

5. The Future Trend

The paper notes a shift in the "gym" (the research community):

From "Pay-to-Play" to "Open Source": In the past, people used expensive, closed AI models to generate training data. Now, they are using open, free models to generate their own practice problems.
From "One-Step" to "Two-Step": We are moving away from just copying (SFT) or just guessing (RL) toward a Hybrid Pipeline where models are first taught by experts, then polished by reward systems.

Summary

This paper is essentially saying: "Don't just memorize the textbook (SFT), and don't just guess in the dark (RL). Do both. Start by learning from the best, then practice until you get it right."

This combination is the secret sauce making AI smarter, more reliable, and better at solving complex real-world problems.

1. Problem Statement

Pre-trained Large Language Models (LLMs) possess broad capabilities but often fail to achieve high accuracy, reliable reasoning, or specific domain proficiency without post-training. Currently, the field relies on two distinct paradigms:

Supervised Fine-Tuning (SFT): Trains models to imitate expert demonstrations (prompt-response pairs) to maximize likelihood.
Reinforcement Learning (RL): Optimizes model behavior based on reward signals derived from human or automated feedback.

While these methods are often treated as separate stages (typically SFT followed by RL, as in RLHF), recent theoretical and empirical evidence suggests they are deeply interconnected. The paper addresses the lack of a unified perspective that systematically compares, integrates, and explains the interplay between SFT and RL. It seeks to clarify when each method is optimal, how they can be combined, and the emerging trends in their application across 2023–2025.

2. Methodology

The study employs a comprehensive survey and theoretical analysis approach, structured as follows:

Theoretical Unification: The authors mathematically demonstrate that SFT can be viewed as a special case of RL. They formulate the SFT objective as an RL problem where the "reward" is an indicator function ( $I\{y = y_i\}$ ) that equals 1 if the model's output matches the ground-truth demonstration. This leads to a unified objective function:
$\max_{\pi} \mathbb{E}[r(x, y)] - \beta \text{KL}(\pi_\theta \parallel \pi_0)$
where $r$ acts as a proxy reward (derived from either explicit rewards in RL or ground-truth matching in SFT) and $\beta$ is a regularization coefficient.
Taxonomy Construction: The paper categorizes existing literature into three main dimensions:
1. Algorithm-centric vs. Data-centric: Analyzing improvements in loss functions (e.g., entropy regularization, policy gradients) versus data selection/synthesis strategies.
2. Integration Frameworks: Reviewing methods that combine SFT and RL, including hybrid training pipelines, single-stage objectives, and mutual enhancement techniques.
3. Application Domains: Examining performance in General QA, Mathematics, Agentic Tasks, and Code Generation.
Empirical Trend Analysis: The authors conducted a systematic analysis of publications from 2023 to 2025. They used a keyword-matching strategy on arXiv preprints to classify papers by domain and training methodology, projecting full-year trends for 2025 based on H1 data.

3. Key Contributions

The paper makes three primary contributions to the field:

Systematic Comparison and Unification: It is the first study to explicitly summarize SFT and RL as complementary tools within a unified theoretical framework. It clarifies that SFT is essentially RL with a specific, sparse reward structure, allowing techniques from one domain (e.g., importance sampling in RL) to be applied to the other.
Hybrid Training Paradigm Analysis: The authors detail how modern research is shifting from sequential SFT $\to$ $\to$ RL pipelines to hybrid approaches. These include:
- Single-stage integration: Combining SFT and RL losses in one objective (e.g., SRFT, UFT).
- Mutual Enhancement: Using SFT to stabilize RL exploration (e.g., prefix sampling, offline demonstrations) and using RL to improve SFT generalization (e.g., reward-weighted SFT, negative reinforcement).
- Dynamic Switching: Algorithms that toggle between SFT and RL based on model performance or task difficulty.
Empirical Trend Identification: The study identifies a rapid shift in the research landscape:
- Hybrid Dominance: The proportion of research using combined SFT+RL pipelines has surged, surpassing pure SFT or pure RL approaches by 2024/2025.
- Data Source Shift: There is a marked transition from relying on API-based labeling (e.g., GPT-4) to using open-weight models to generate synthetic training data and rollouts.
- Domain Expansion: Significant growth in applications for mathematical reasoning, agentic planning, and code generation, with math and code showing the highest growth rates.

4. Results and Findings

Theoretical Insight: The gradient analysis confirms that optimizing with both SFT and RL objectives collapses into a unified RL objective. This implies that "tricks" developed for one (like entropy regularization or KL clipping) are applicable to the other.
Performance Trade-offs:
- SFT is superior for initial alignment, stability, and memorizing expert patterns but suffers from distribution shift and limited generalization to novel problems.
- RL excels at exploration, step-wise reasoning, and generalization but risks entropy collapse, reward hacking, and instability without strong initialization.
- Hybrid methods consistently outperform standalone methods by leveraging SFT for stability and RL for optimization.
Application Specifics:
- Math: Hybrid approaches using step-wise rollouts and verification (e.g., GRPO, Critique-DPO) have significantly improved reasoning capabilities.
- Agents: Hierarchical RL combined with SFT is critical for long-horizon planning and action selection.
- Code: RL with executable feedback (unit tests) combined with SFT on open-source code has become the standard for code generation and editing.
Trend Data (2023–2025):
- Research volume in Code tasks grew by ~272% (2023 to 2024), and Math by ~123%.
- The proportion of papers using Hybrid (SFT+RL) training rose from 20% in 2023 to ~74% in 2024, becoming the dominant paradigm.
- Reliance on API-based models for data generation dropped from 32% to 11%, while open-weight model usage doubled.

5. Significance

This paper serves as a foundational reference for the next generation of LLM post-training. Its significance lies in:

Bridging the Gap: It dismantles the artificial barrier between SFT and RL, providing a coherent theoretical basis for why and how they should be integrated.
Guiding Future Research: It highlights the urgent need for sample- and compute-efficient methodologies, as current hybrid pipelines are resource-intensive. It also points to the challenge of training under sparse or indirect reward signals (e.g., user feedback, self-evaluation).
Standardizing Practices: By documenting the shift toward open-weight, model-generated datasets and hybrid pipelines, it outlines the emerging "best practices" for scalable and generalizable LLM alignment.

In conclusion, the study argues that the future of LLM alignment lies not in choosing between SFT or RL, but in integrating them into flexible, hybrid frameworks that leverage the stability of imitation learning and the adaptability of reward-driven optimization.