Technical Summary: RASFT (Rollout-Adaptive Supervised Fine-Tuning for Reasoning)

1. Problem Statement

Supervised Fine-Tuning (SFT) is the standard approach for adapting Large Language Models (LLMs) to reasoning tasks by imitating offline expert demonstrations. However, the authors identify a critical limitation in this paradigm: reasoning is not a simple path imitation task. A single problem often admits multiple valid solution paths, yet standard SFT treats a single offline expert trajectory as the rigid target behavior.

This rigid imitation can lead to two issues:

1. Overfitting to Surface Forms: The model may memorize the specific demonstration rather than learning the underlying reasoning distribution.
2. Suppression of Latent Ability: Pretrained LLMs often possess rich, latent reasoning distributions. Over-imitating a single expert path can overwrite these priors, weakening the model's own ability to generate diverse, correct reasoning trajectories.

Existing SFT variants (e.g., DFT, ASFT, ProFiT) attempt to mitigate this by rescaling learning signals or anchoring to reference models, but they remain largely "demonstration-internal." They optimize based on signals within the offline trajectory rather than calibrating supervision based on the current policy's actual ability to solve the problem.

2. Methodology: RASFT

The authors propose Rollout-Adaptive Supervised Fine-Tuning (RASFT), a policy-aware framework that dynamically balances expert guidance with self-generated reasoning based on the model's current performance.

Core Components

1. Local Candidate Pool Construction:
   For each problem prompt $x_i$, RASFT constructs a pool $G_i$ containing:

   • The offline expert trajectory ($y^e_i$).
   • Verified correct trajectories generated by the current policy ($\pi_\theta$) via on-policy rollouts.
   • Note: Incorrect rollouts are filtered out from the optimization targets but used to estimate difficulty.

2. Problem-Level Solvability Estimation ($\zeta_i$):
   The model generates $K$ rollouts for a given prompt. A task-specific verifier (e.g., final-answer matching for math, assertion tests for code) determines correctness. The solvability index is calculated as the success rate:
   $$\zeta_i = \frac{1}{K} \sum_{k=1}^{K} c_{i,k}$$
   where $c_{i,k} \in \{0, 1\}$ indicates correctness. This score reflects how well the current policy can solve the problem.

3. Adaptive Weighting:
   RASFT assigns weights to trajectories in the candidate pool based on $\zeta_i$:

   • Expert Trajectory Weight ($W_e$): Dynamically adjusted. When $\zeta_i$ is low (model struggles), the expert trajectory receives a higher weight to provide strong corrective guidance. When $\zeta_i$ is high (model is reliable), the weight is attenuated to prevent rigid fitting and allow the model's own reasoning to dominate.
   • Self-Generated Trajectory Weight ($W_r$): Fixed for verified correct rollouts.

4. Policy-Regularized Objective:
   To prevent excessive drift from the pretrained distribution, RASFT introduces a clipped inverse ratio between a frozen reference model ($\pi_{ref}$) and the current policy ($\pi_\theta$). The final loss function combines:

   • Normalized trajectory weights ($\bar{W}_{i,j}$).
   • The clipped inverse ratio ($Clip(r_{i,j}(\theta))$).
   • Token-level confidence ($p_{\theta}$).

   The objective minimizes a weighted sequence-to-sequence loss:
   $$\mathcal{L}(\theta) = -\mathbb{E}_i \left[ \sum_{y_{i,j} \in G_i} \sum_{t=1}^{|y_{i,j}|} \omega_{i,j,t} \log p_{\theta, i,j,t} \right]$$
   where $\omega_{i,j,t}$ is the compound weight.

3. Key Contributions

The paper summarizes its contributions as follows:

• Identification of Limitation: It highlights that existing SFT variants lack policy-awareness, failing to distinguish when expert imitation is necessary versus when it becomes rigid path fitting.
• Framework Proposal: RASFT is introduced as a rollout-adaptive SFT framework that uses verified on-policy rollouts to estimate problem solvability, dynamically balancing expert guidance with self-generated correct trajectories.
• Empirical Validation: Extensive experiments demonstrate that RASFT outperforms standard SFT, SFT variants (DFT, ASFT, ProFiT), and representative Reinforcement Learning (RL) methods (GRPO, LUFFY) across mathematical and code reasoning tasks.

4. Experimental Results

Experiments were conducted on six mathematical reasoning benchmarks (MATH-500, Minerva, OlympiadBench, AIME 2024/2025, AMC 2023) and two code reasoning benchmarks (HumanEval, MBPP) using models from the Qwen2.5 and Llama-3.2 families.

• Performance Gains: RASFT consistently achieved the best average performance.
  • Math: On Qwen2.5-Math-1.5B, it achieved a relative gain of 10.9% (25.00 $\to$ 27.72) over the best baseline.
  • Code: On Llama-3.2-3B, it achieved a 26.9% gain (24.93 $\to$ 31.63).
  • Comparison with RL: RASFT improved average math accuracy by 15.9% (20.25 $\to$ 23.47) compared to GRPO.
• Ablation Studies:
  • Difficulty Adaptivity: Removing adaptive weighting caused a drop in average score from 27.72 to 25.53, confirming that static balancing is suboptimal.
  • On-Policy Rollouts: Removing rollouts (reverting to static expert weights) led to uniform degradation, highlighting the necessity of online feedback.
  • Inverse Ratio: Removing the inverse ratio reduced performance, confirming its role in preventing overfitting to offline data and preserving pretrained priors.
  • Transferability: RASFT improved performance when integrated with standard SFT and ProFiT, suggesting it is complementary to token-level rescaling methods.
• Training Dynamics: RASFT maintained stable training objectives and lower parameter-update density in later stages compared to other methods, indicating better preservation of pretrained reasoning priors.

5. Significance and Claims

The paper claims that RASFT represents a shift from static demonstration fitting to policy-adaptive supervision. Its significance lies in:

1. Bridging SFT and RL: It incorporates the benefits of online exploration (rollouts) and adaptive weighting typically found in RL methods but retains the stability and simplicity of the SFT framework.
2. Dynamic Calibration: It successfully calibrates the strength of expert supervision based on the model's actual capability to solve a specific problem, avoiding the suppression of the model's own reasoning distribution.
3. Robustness: The method is robust across different model scales and task domains, particularly showing significant gains on difficult benchmarks (e.g., AIME, OlympiadBench) where rigid imitation fails.

Limitations Acknowledged by the Authors:

• Computational Cost: RASFT is more expensive than standard SFT due to the periodic generation and verification of on-policy rollouts.
• Verification Dependency: The method relies on reliable automatic verification signals (e.g., final answers for math, test cases for code). Extending it to open-ended reasoning tasks without clear correctness signals remains a challenge.