Technical Summary: Rethinking Expert Trajectory Utilization in LLM Post-training for Mathematical Reasoning

Problem Statement

The post-training of Large Language Models (LLMs) for mathematical reasoning is currently dominated by two paradigms: Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL). While SFT relies heavily on expert trajectories (prompt-solution pairs) to instill reasoning priors via imitation, RL methods (e.g., GRPO, DAPO) typically leverage reward signals to explore reasoning paths without direct expert trajectory utilization.

Despite the consensus that both are necessary, the optimal mechanism for utilizing expert trajectories to maximize the final performance ceiling remains unresolved. The field faces a "paradigm dilemma":

1. Synchronized Approaches (Syn-SFT-RL): Recent methods (e.g., UPT, SRFT, LUFFY) integrate imitation loss directly into the RL loop, claiming superior efficiency. However, these are often constrained by limited SFT data scales (e.g., ~46K samples) and lack rigorous comparison against baselines trained on significantly larger datasets.
2. Sequential Approaches (SFT-then-RL): The industry standard involves a sequential pipeline. However, the principles governing this approach are largely empirical. Critical questions remain unanswered regarding the optimal timing for switching from SFT to RL, the impact of data scale versus difficulty, and whether the "Less is More" hypothesis (using minimal high-quality data) holds true for maximizing the ultimate reasoning ceiling.

Methodology: The Plasticity-Ceiling Framework

To address these gaps, the authors propose the Plasticity-Ceiling Framework, a unified analytical approach that decomposes the theoretical post-training performance ceiling ($A_{post}$) into two measurable components:

1. SFT Performance ($P_{sft}$): The realized performance achieved after the SFT phase given a specific compute allocation ($x_{sft}$).
2. RL Plasticity ($P_{L_{rl}}$): The maximum potential improvement achievable via subsequent RL, defined as the difference between the asymptotic ceiling and the SFT baseline ($A_{post} - P_{sft}$).

The framework models the post-training performance as a function of RL compute using a sigmoidal law, allowing for the quantitative decomposition of the final ceiling. This enables the authors to systematically evaluate how SFT data properties (scale and difficulty) and training dynamics influence both the foundation ($P_{sft}$) and the headroom for improvement ($P_{L_{rl}}$).

Experimental Setup

The study conducts extensive benchmarks on Qwen2.5-7B and Llama3.2-3B across six mathematical benchmarks (GSM8K, OlympiadBench, Minerva, MATH, AIME24/25) and out-of-domain (OOD) tasks (GPQA-Diamond, ARC-C).

• Datasets: The authors curate SFT datasets ranging from 1K to 889K unique samples, derived from distilled DeepSeek-R1 outputs. These include scale variants (S1K, Uniform102K, SFT889K) and difficulty variants (Easy102K, Uniform102K, Hard102K).
• Baselines: The study compares Pure-SFT, Pure-RL (GRPO, DAPO), Syn-SFT-RL (LUFFY, SRFT, UPT, Prefix-RFT), and the Sequential SFT-then-RL pipeline.
• Metrics: Performance is measured via pass@1 accuracy, with compute costs normalized to FLOPs (exaFLOPs).

Key Results

1. Paradigm Selection: Sequential Dominance

The Sequential SFT-then-RL pipeline is established as the superior standard.

• Syn-SFT-RL Limitations: Synchronized methods exhibit training instability and premature convergence. They are highly sensitive to model priors and fail to scale effectively when compared to sequential baselines optimized on larger datasets.
• The "Slow but High" Advantage: Pure-SFT demonstrates continuous improvement with extensive training, reaching a peak significantly higher than the ceilings of Pure-RL or Syn-SFT-RL methods. Crucially, transitioning to RL after SFT saturation unlocks further gains, achieving the highest performance frontier (e.g., 78.1 points on Qwen2.5-7B vs. 74.3 for Pure-RL).

2. Optimal Transition Timing: SFT Saturation

The optimal strategy for switching from SFT to RL is to wait until the Stable Regime or Mild Overfitting Regime of the SFT validation loss.

• Premature Switching: Switching during the Adaptive Regime (underfitting) yields suboptimal ceilings because the model lacks the foundational competence required for RL to recover.
• Severe Overfitting: Continuing SFT into the Severe Overfitting Regime causes a collapse in RL plasticity, detrimental to the final ceiling.
• Generalization: Training SFT to saturation establishes a robust foundation for Out-of-Domain (OOD) generalization, contradicting claims that intensive SFT degrades generalization.

3. Data Properties: Scale and Difficulty

The study refutes the "Less is More" hypothesis in the context of SFT-then-RL scaling.

• Scale Dominates: Data scale is the primary determinant of the final performance ceiling. While small datasets (e.g., S1K) achieve rapid initial gains, they saturate prematurely. Large-scale datasets (e.g., SFT889K) provide both higher realized SFT performance and greater preserved RL plasticity, leading to a significantly higher ceiling.
• Difficulty as a Multiplier: Trajectory difficulty acts as a performance multiplier. Harder data (Hard102K) yields higher SFT performance and maintains comparable plasticity to easier data, making it the superior choice when scale is fixed.
• Predictive Indicator: The Minimum Validation Loss of the SFT phase serves as a reliable, robust indicator for the final post-training ceiling, strongly correlating with the ultimate performance ($A_{post}$).

Significance and Contributions

The paper claims to transform expert trajectory optimization from empirical guesswork into a systematic, predictable process. Its primary contributions are:

1. The Plasticity-Ceiling Framework: A theoretical mechanism that decomposes post-training performance into realized SFT gains and subsequent RL plasticity, providing a unified view for paradigm selection.
2. Rigorous Benchmarking: Establishing the Sequential SFT-then-RL pipeline as the rigorous standard for stability and performance, outperforming synchronized approaches and pure methods.
3. Actionable Scaling Guidelines:
   • Timing: Initiate RL at SFT saturation (Stable Regime).
   • Data Strategy: Prioritize data scale over difficulty to maximize the ceiling; use difficulty to optimize within limited scale constraints.
   • Monitoring: Use Minimum Validation Loss as a proxy for the final performance ceiling.

The authors conclude that these findings provide actionable guidelines for extracting maximum value from expert trajectories, specifically for mathematical reasoning tasks using R1-style trajectories. They acknowledge limitations regarding generalization to non-verifiable tasks (e.g., creative writing) or alternative trajectory styles, leaving these as open questions for future work.