Technical Summary: Restoring the Sweet Spot via Pass-Rate Weighted Self-Distillation

1. Problem Statement

Large Language Models (LLMs) are increasingly trained using Reinforcement Learning with Verifiable Rewards (RLVR) for domains like code and mathematics. While RLVR methods like Group Relative Policy Optimization (GRPO) have shown success, they suffer from a "credit-assignment bottleneck" where a single scalar reward per attempt fails to indicate where or why a model failed.

Self-Distillation Policy Optimization (SDPO) addresses this by using the model itself as a teacher, conditioned on privileged information (e.g., runtime errors or correct solutions), to provide dense, token-level supervision. However, the authors identify a critical trade-off:

• GRPO possesses an implicit "difficulty awareness." Its group-relative advantage normalization naturally concentrates learning on a "sweet spot" of intermediate-difficulty questions (where the pass rate $p \approx 0.5$), suppressing learning on questions that are too easy ($p \approx 1$) or too hard ($p \approx 0$).
• SDPO, while providing superior dense credit assignment, lacks this difficulty awareness. Its KL-based loss treats all questions equally, diluting the learning signal on the most informative intermediate-difficulty samples and failing to suppress uninformative all-correct or all-incorrect groups.

The core problem is that SDPO sacrifices the beneficial "sweet spot" concentration of GRPO to gain dense credit assignment, resulting in suboptimal learning efficiency.

2. Methodology: SC-SDPO

The authors propose Scale-Consistent SDPO (SC-SDPO), a variant that restores the difficulty-aware "sweet spot" to SDPO without sacrificing its dense credit assignment or requiring external teachers.

Theoretical Foundation

The paper extends the learnability framework of Bae et al. [2026] to normalized rewards.

1. Raw vs. Normalized Rewards: Under raw binary rewards, policy improvement is lower-bounded by the variance $p(1-p)$. However, GRPO uses group-relative normalization: $\hat{A}_i = (r_i - \bar{r}) / \text{std}(\{r_i\})$.
2. The Scaling Factor: The authors demonstrate that this normalization absorbs the $p(1-p)$ variance term, equalizing the leading-order learnability across questions. Consequently, the residual per-question gradient magnitude in GRPO scales as $\sqrt{p(1-p)}$, not $p(1-p)$.
3. The Ideal Objective: To combine SDPO's dense token-level advantages with GRPO's difficulty modulation, the ideal per-question gradient should be scaled by a factor of $[\hat{p}(1-\hat{p})]^{1/2}$.

Algorithmic Implementation

SC-SDPO modifies the SDPO loss function by introducing a per-question scalar weight $\bar{w}_j$:
$$\mathcal{L}_{\text{SC-SDPO}}(\theta) := \sum_{j=1}^{M} \bar{w}_j \sum_{i \in G_j} \sum_{t} \text{KL}(\pi_\theta(\cdot | x_j, y_{i,<t}) \parallel \text{sg}(\pi_\theta(\cdot | x_j, f_i, y_{i,<t})))$$

Key components of the weighting scheme:

• Weight Function: The unnormalized weight is $w_j = [\hat{p}_j(1-\hat{p}_j)]^\alpha$, where $\hat{p}_j$ is the empirical pass rate for question $j$ within a batch. The authors set $\alpha = 1/2$ based on the theoretical derivation.
• Batch-Adaptive Normalization: To prevent training instability as the distribution of pass rates shifts during training, weights are normalized within the batch: $\bar{w}_j = w_j / \text{mean}(\{w_l\}_{l \in S})$, where $S$ is the set of non-degenerate questions.
• Zero-Cost Byproduct: Since SDPO is an on-policy algorithm that already samples multiple rollouts per question, the pass rate $\hat{p}_j$ is a free byproduct of the existing training loop. No additional rollouts or offline passes are required.

3. Key Contributions

1. Theoretical Analysis: The paper identifies that GRPO's advantage normalization absorbs the variance term $p(1-p)$, leaving $\sqrt{p(1-p)}$ as the natural scaling factor for gradient magnitude. This corrects a previous assumption that the scaling should be linear with variance.
2. SC-SDPO Algorithm: A simple, single-scalar modification to SDPO that restores difficulty awareness. It dynamically weights questions based on their evolving pass rates, creating an implicit curriculum that tracks model competence.
3. Empirical Validation: Extensive experiments demonstrating that combining dense credit assignment (SDPO) with difficulty-aware weighting (SC-SDPO) yields superior performance compared to using either mechanism alone.

4. Experimental Results

Experiments were conducted on Qwen3-8B and OLMo-3-7B across scientific reasoning (Chemistry, Physics, Biology, Materials Science) and tool-use benchmarks.

• Performance Gains:
  • On Qwen3-8B, SC-SDPO ($\alpha=0.5$) improved over the unweighted SDPO baseline by +3.2 (mean@16) and +4.3 (maj@16) on average.
  • On OLMo-3-7B, it improved by +1.8 (mean@16) and +3.0 (maj@16).
  • SC-SDPO consistently outperformed GRPO, SDPO, and other baselines (PACED, Hard Filter, SRPO).
• Exponent Analysis: The choice of $\alpha=0.5$ (scale-consistent) consistently outperformed $\alpha=1$ (variance-based), validating the theoretical claim that $\alpha=1$ over-suppresses the learning signal.
• Training Dynamics:
  • SC-SDPO maintained stable gradient norms similar to the unweighted SDPO baseline, whereas static weighting (PACED) caused gradient spikes and hard filtering suppressed gradients too aggressively.
  • The method successfully retained a higher fraction of intermediate-difficulty samples ("sweet spot") during early training, accelerating skill acquisition.
  • Response lengths remained higher than the SDPO baseline, suggesting the model was encouraged to elaborate on intermediate-difficulty questions rather than collapsing to short answers.

5. Significance and Claims

The paper claims that SC-SDPO resolves the fundamental tension between dense credit assignment and difficulty awareness in LLM post-training. By deriving the correct scaling factor ($\sqrt{p(1-p)}$) through theoretical analysis of normalized rewards, the authors provide a "simple prescription" that restores the "sweet spot" to self-distillation.

The significance lies in:

• Efficiency: It achieves state-of-the-art results without requiring external teacher models or additional computational overhead (zero-cost pass-rate estimation).
• Robustness: The batch-adaptive normalization ensures stable training dynamics, avoiding the pitfalls of static or frozen weighting schemes.
• Generalizability: The method works across different model families (Qwen and OLMo) and task types (reasoning and tool use).

The authors remain modest regarding limitations, noting that the method relies on binary rewards and discrete pass-rate granularity (dependent on the number of rollouts $G$), and its efficacy on open-ended generation tasks or significantly larger model scales requires further validation.