Technical Summary: Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning

1. Problem Statement

The post-training of reasoning Large Language Models (LLMs) typically follows a two-stage paradigm: an offline Supervised Fine-Tuning (SFT) stage to produce an initial checkpoint, followed by an online Reinforcement Learning (RL) stage to further enhance performance. A prevailing assumption in current pipelines is that optimizing the SFT stage for maximum offline accuracy (e.g., maximizing next-token likelihood or minimizing loss on the training distribution) will naturally yield a stronger initialization for subsequent RL.

However, the authors demonstrate that this assumption is flawed. They observe that models initialized from "stronger" SFT checkpoints (those with superior offline performance) can significantly underperform those initialized from "weaker" checkpoints after undergoing identical RL training. This phenomenon is attributed to a distribution mismatch between the offline and online stages:

• Offline SFT: The model learns from data sampled from a behavior policy ($\pi_\beta$), often a different model or a static dataset.
• Online RL: The model learns from rollouts generated by its own evolving target policy ($\pi_\theta$).

Standard SFT applies uniform token-level supervision based on the behavior policy's distribution. In long-horizon reasoning tasks, this can reinforce transition patterns (e.g., dead-end continuations) that are highly probable under the behavior policy but implausible under the target policy. Consequently, the model learns to associate prefixes with continuations it will rarely revisit during online RL, leading to a suboptimal initialization that hinders downstream learning.

2. Methodology: PEAR

To address this mismatch, the authors propose PEAR (Policy Evaluation–inspired Algorithm for Offline Learning Loss Reweighting). PEAR is an SFT-stage method that reweights the training loss to better align the offline objective with the target policy's distribution, effectively preparing the model for the online RL phase.

Core Mechanism

PEAR utilizes importance sampling to correct the distribution shift between the behavior policy ($\pi_\beta$) and the target policy ($\pi_\theta$). Instead of treating all tokens in the offline dataset equally, PEAR reweights the loss of each token based on the likelihood ratio between the target policy and the behavior policy.

The objective function is modified as follows:
$$L_{PEAR}(\theta) \triangleq \mathbb{E}_{(x,y) \sim D} \left[ \sum_{t=1}^{T} \text{sg}[\hat{G}_t] \cdot \ell_\theta(x, y_{<t}, y_t) \right]$$
Where:

• $\ell_\theta$ is the standard per-token loss (e.g., NLL or KL divergence).
• $\text{sg}[\cdot]$ denotes a stop-gradient operator, treating the weight as a fixed coefficient.
• $\hat{G}_t$ is the numerically stabilized importance weight for token $t$.

Weighting Variants

PEAR offers three granularities for computing the weights $\hat{G}_t$:

1. Sequence-Level Weighting: Applies a single global importance ratio ($w_{1:T} = \prod_{t=1}^T \frac{\pi_\theta(y_t|x, y_{<t})}{\pi_\beta(y_t|x, y_{<t})}$) to all tokens in a trajectory. This assumes the entire sequence's plausibility under the target policy is uniform.
2. Token-Level Weighting (Suffix-Based): Computes weights based on the suffix importance ratio. For a token $y_t$, the weight is determined by the likelihood of the remaining continuation ($y_{>t}$) under the target policy relative to the behavior policy:
   $$G_t = \gamma^{T-t} \prod_{j=t+1}^{T} \frac{\pi_\theta(y_j|x, y_{<j})}{\pi_\beta(y_j|x, y_{<j})}$$
   This down-weights tokens that lead to continuations the target policy is unlikely to generate, focusing learning on prefixes compatible with the target policy's future behavior.
3. Block-Level Weighting: To mitigate the high variance inherent in long-horizon multiplicative importance weights, the sequence is partitioned into blocks. Weights are computed and applied uniformly within each block, trading some granularity for stability.

Additional Features

• Negative Examples: PEAR can incorporate verified failure trajectories by applying a repulsive term, pushing the model away from negative continuations while maintaining policy consistency.
• Black-Box Adaptation: When the exact behavior policy $\pi_\beta$ is unavailable (e.g., closed-source data), PEAR can utilize a proxy model or an ensemble of proxies to approximate the behavior policy's log-probabilities.
• Numerical Stabilization: The method employs log-space computation and clipping of log-ratios to prevent numerical instability during the calculation of long product sequences.

3. Key Contributions

• Empirical Demonstration of Offline-Online Decoupling: The paper provides controlled experiments showing that offline performance is an unreliable proxy for post-RL performance. Stronger SFT objectives can lead to weaker final RL results due to distribution mismatch.
• PEAR Algorithm: The introduction of a practical, plug-in reweighting scheme that uses off-policy evaluation (OPE) principles to align offline SFT with online RL goals.
• Comprehensive Evaluation: Extensive testing across multiple model families (Qwen2.5, Qwen3, DeepSeek-distilled) and sizes (0.6B to 8B) on verifiable reasoning tasks (logic games, math benchmarks).
• Analysis of Parameter Drift: The authors show that PEAR-initialized models undergo less parameter drift during subsequent RL training, suggesting that PEAR reduces the need for the RL phase to correct offline-to-online misalignments.

4. Experimental Results

The authors conducted experiments on synthetic logic puzzles (SynLogic, Enigmata) and mathematical reasoning benchmarks (AIME-2024/2025, AMC-2023, MATH-500).

• Performance Gains: PEAR consistently outperforms canonical SFT and other baselines (including KL-regularized SFT, TALR, and various probability-based objectives) after RL.
  • On AIME-2025, PEAR-initialized models achieved Pass@8 gains of up to 30 percentage points compared to standard SFT (e.g., increasing from 5% to 35% on DeepSeek-Distill-Qwen-1.5B).
  • On SynLogic logic games, PEAR showed Pass@1 improvements of up to +25 percentage points (e.g., 13.1% to 38.3% on Qwen3-1.7B-Base).
• Generalization: The benefits of PEAR transfer across different RL task distributions (e.g., training on SynLogic and RL on Enigmata) and different RL algorithms (GRPO and DAPO).
• Black-Box Robustness: Even when using proxy models to approximate the behavior policy, PEAR significantly outperforms standard SFT, with strong proxies matching white-box performance.
• Stability: PEAR-initialized models exhibit smaller parameter drift (measured by Normalized Spectrum Shift) during the online RL phase compared to models initialized with standard SFT, indicating a more efficient learning trajectory.

5. Significance and Claims

The paper argues that the goal of the offline SFT stage should not merely be to maximize offline accuracy, but to prepare the policy for the subsequent online RL.

• Holistic Post-Training: The authors claim that designing SFT objectives with downstream RL in mind is crucial. Optimizing for offline metrics in isolation can be counterproductive, as it may reinforce behaviors that are incompatible with the target policy's exploration and optimization dynamics.
• Correcting Mismatch: PEAR serves as a concrete instantiation of the principle that offline training must correct for the behavior-target occupancy mismatch. By down-weighting implausible continuations and up-weighting those compatible with the target policy, PEAR creates a more effective initialization.
• Practical Impact: The results suggest that simple reweighting strategies based on off-policy evaluation can yield substantial improvements in reasoning capabilities without requiring complex architectural changes or additional data collection, provided the behavior policy probabilities are accessible (or approximated).

The authors conclude that "a good offline stage should not optimize for offline accuracy, but prepare the policy for the RL that follows," positioning PEAR as a step toward more holistic LLM post-training pipelines.