Technical Summary: Strategy-Guided Policy Optimization (SGPO) for LLM Reasoning

1. Problem Statement

Current methods for distilling reasoning capabilities from strong to weak Large Language Models (LLMs) predominantly rely on trajectory-level imitation. Whether through Supervised Fine-Tuning (SFT) on expert traces or hybrid policy optimization, these approaches train student models to reproduce specific solution sequences generated by experts. The paper argues that this unit of knowledge transfer is flawed: it encourages the memorization of instance-specific patterns rather than the acquisition of transferable problem-solving skills. Consequently, models trained this way struggle to generalize to novel problems because they learn what to answer rather than how to reason.

The core challenge addressed is how to shift the distillation target from concrete solution trajectories to reusable problem-solving strategies that explain why specific steps were chosen, without requiring the model to rely on external hints during inference.

2. Methodology: Strategy-Guided Policy Optimization (SGPO)

The authors propose SGPO, a framework that replaces instance-level trajectory imitation with reusable strategy distillation. The framework operates by extracting structured strategy descriptions from strong-model responses and using them to reshape the student model's reasoning distribution.

2.1 Core Components

For each training problem $q$, SGPO constructs two trajectory groups:

1. Autonomous Group: Sampled from the current policy $\pi_\theta(\cdot | q)$.
2. Strategy-Guided Group: Sampled from $\pi_\theta(\cdot | q, s)$, where $s$ is a concise natural-language strategy description (specifying problem type, approach, and procedural steps) extracted from a strong model, but containing no computations or final answers.

The framework addresses two key questions: How to distill and When to distill.

2.2 How to Distill: Token-Level Forward-KL Objective

Instead of directly imitating the guided trajectory (which would collapse back into SFT), SGPO distills the distributional shift induced by strategy conditioning.

• Forward-KL Divergence: The method minimizes the Kullback-Leibler (KL) divergence where the strategy-guided distribution is the reference and the unguided distribution is the target ($D_{KL}(\pi_{guided} || \pi_{unguided})$).
• Selectivity: Unlike Reverse KL, which would collapse the policy onto a single guided mode, Forward KL encourages the unguided policy to cover the guided mode while preserving existing alternatives discovered through autonomous exploration.
• Mechanism: The objective naturally concentrates optimization pressure on tokens where the next-token distribution shifts significantly under strategy conditioning (e.g., choosing a specific mathematical method). These correspond to strategy-critical decision points, whereas routine linguistic tokens exhibit low KL divergence. This provides a selective distillation signal without requiring explicit token-level annotations.

2.3 Stability and Efficiency: Proximal Constraints

To ensure training stability and efficiency, SGPO employs three levels of proximal constraints:

1. Trajectory Level (Reachable Target Selection): Instead of averaging over all correct guided trajectories, the method selects the single guided trajectory most reachable by the current unguided policy. This minimizes the distributional gap the model must bridge.
2. Token Level (KL Clipping): Token positions where the divergence between guided and unguided distributions exceeds a threshold $\delta$ are masked. This prevents outlier positions from dominating the optimization and causing entropy collapse.
3. Instance Level (Adaptive Weighting): A dynamic weight $\alpha(q)$ modulates the strength of distillation based on the marginal benefit of strategy guidance.
   • If the autonomous pass rate is low and the guided pass rate is high, $\alpha(q)$ is large, strengthening guidance.
   • As the model's autonomous competence grows, $\alpha(q)$ decreases, naturally transitioning training from strategy-driven improvement to autonomous optimization without manual scheduling.

2.4 Overall Objective

The total loss combines autonomous Group Relative Policy Optimization (GRPO) with the proximal strategy distillation term:
$$\mathcal{L}(\theta; q) = \mathcal{L}_{GRPO}(\theta; q) + \alpha(q) \mathcal{L}^{clip}_{KD}(\theta; q)$$

3. Key Contributions

• Paradigm Shift: Moves reasoning distillation from imitating specific solution trajectories to internalizing reusable problem-solving strategies.
• Forward-KL Distillation: Introduces a token-level forward-KL objective that selectively transfers strategic knowledge by focusing on probability shifts at decision points, outperforming direct trajectory imitation.
• Adaptive Framework: Proposes a training dynamic where distillation strength automatically scales with the model's competence, eliminating the need for manual curriculum scheduling.
• Complementary Scaling: Demonstrates that strategy distillation benefits scale with base model capability, suggesting a minimum reasoning competence is required to effectively internalize strategic guidance.

4. Experimental Results

Experiments were conducted on four mathematical reasoning benchmarks (MATH500, AMC 23, OlympiadBench, AIME 24) using two model families: Qwen2.5 (1.5B and 7B) and Llama-3.2 (8B).

• Performance: SGPO consistently outperformed strong baselines, including SFT, on-policy RL (SFT+GRPO), and hybrid-policy methods (HPT, LUFFY).
  • On Qwen2.5-7B-Instruct, SGPO improved the average score by 2.2 points over the strongest baseline (LUFFY) and by 9.9 points over the base model.
  • On Qwen2.5-1.5B-Instruct, the average score improved by 6.7 points over the base model.
• Ablation Studies:
  • Removing strategy distillation reduced performance, confirming the value of strategy-level transfer over pure RL.
  • Removing autonomous GRPO led to entropy collapse and performance stagnation, highlighting the necessity of maintaining exploration diversity.
  • Removing proximal constraints caused the largest performance drop (6.8 points), indicating their critical role in stability.
• Analysis:
  • Selectivity: The forward-KL signal peaked at strategy-critical steps (e.g., determining the least common denominator) while remaining low for routine tokens, unlike the monotonically declining loss of direct SFT.
  • Scaling: Gains from SGPO were larger for stronger base models, suggesting that strategy distillation is complementary to foundational reasoning competence.

5. Significance and Claims

The paper claims that transferring knowledge at the strategy level is a more effective direction for reasoning distillation than trajectory-level imitation.

• Generalization: By internalizing how to reason (strategies) rather than what to answer (trajectories), models acquire transferable skills that generalize better to novel problems.
• Inherent Selectivity: The proposed forward-KL objective acts as an automatic mechanism to identify and amplify strategy-critical tokens, removing the need for manual annotation of reasoning steps.
• Efficiency: The adaptive weighting mechanism ensures that computational resources are focused on instances where strategy guidance provides the highest marginal benefit, leading to faster convergence and higher performance ceilings.

The authors conclude that while improved reasoning capabilities carry potential risks, the specific focus on mathematical reasoning poses minimal direct risk, and the framework offers a robust path for enhancing LLM reasoning through strategy distillation.