Technical Summary: Skill-Conditioned Gated Self-Distillation for LLM Reasoning

1. Problem Statement

Large Language Model (LLM) reasoning is frequently enhanced via Reinforcement Learning with Verifiable Rewards (RLVR), such as Group Relative Policy Optimization (GRPO). While RLVR requires only a verifier and no token-level annotations, it suffers from sparse, delayed, and nearly uniform reward signals across entire reasoning trajectories. On-policy self-distillation (SD) addresses this by using a stronger "teacher" context to provide dense token-level supervision. However, existing SD methods typically rely on trusted privileged information (PI), such as reference answers or successful traces, which may not be available in weak supervision regimes (e.g., when only a verifier exists).

The paper investigates whether PI can instead be derived from an experience-based skill bank—a collection of compact, reusable natural-language principles and mistake patterns. The core challenge is that retrieved skills are not inherently trusted; they may be irrelevant, stale, or misleading for a specific problem. Furthermore, blindly imitating a teacher conditioned on a potentially wrong skill can degrade performance. The central question is: Can experience-derived skills serve as training-only privileged information without assuming every retrieved skill is trusted?

2. Methodology: Skill-Conditioned Gated Self-Distillation (SGSD)

The authors propose SGSD, a framework that treats skill-based self-distillation as a teacher hypothesis validation problem rather than unconditional imitation. The method operates on a single policy model $\pi_\theta$ acting as both student and teacher under different contexts.

2.1 Structured Skill Bank Construction

SGSD maintains a structured bank $B = \{G, E\}$ containing:

• General Skills ($G$): Reusable reasoning principles (what to do).
• Common Mistakes ($E$): Error patterns, causes, and avoidance strategies (what to avoid).
  The bank is initialized via a "cold-start" process where successful traces are summarized into skills and failed traces into mistakes. During training, the bank is updated online by merging new candidates from recent rollouts.

2.2 Skill-Conditioned Multi-Teacher Pool

For a given problem $x$, the system retrieves top-$K$ skill-mistake pairs. Each pair $(g_k, e_k)$ is concatenated with the problem to form a unique teacher context $c_k(x) = g_k \oplus e_k \oplus x$.

• Student: Samples a rollout $y$ using only the plain prompt $x$.
• Teachers: A pool of $K$ teachers, each sharing the same policy weights (stop-gradient) but conditioned on a specific retrieved skill-mistake pair. All teachers score the same student rollout.

2.3 Outcome-Validated Teacher Polarity

Instead of assuming all teachers are correct, SGSD validates their stance against the verifier outcome $r \in \{-1, 1\}$.

1. Gap Calculation: Compute the log-probability gap $\Delta^{(k)}_t$ between teacher $k$ and the student for each token.
2. Robust Support Estimation: Calculate a robust support score $\tilde{a}_k$ by masking non-reasoning tokens, clipping extreme gaps, and averaging.
3. Polarity Inference: Determine the teacher's polarity $\rho_k$ based on the alignment between the teacher's support ($\tilde{a}_k$) and the outcome ($r$):
   • Helpful: Teacher supports a success or suppresses a failure ($\rho_k = +1$).
   • Misleading: Teacher supports a failure or suppresses a success ($\rho_k = -1$).
   • Uncertain: If the support score is near zero, the teacher is ignored ($\rho_k = 0$).

2.4 Robust Gated Distillation Objective

The final loss function aggregates the weighted, polarity-aware disagreements:
$$L_{SGSD}(x) = \sum_{k=1}^{K_x} \alpha_k(x) \rho_k \bar{\ell}^{(k)}$$
Where $\bar{\ell}^{(k)}$ is a gated loss $\ell_{gate}(\Delta)$ designed to:

• Be near-zero when teacher and student agree (negligible gaps).
• Grow for moderate disagreements (informative region).
• Saturate for extreme mismatches to prevent outlier logits from dominating the gradient.
  The derivative of this loss ensures the update is bounded and behaves like a local reverse-KL approximation in the informative region while suppressing extreme signals.

3. Key Contributions

1. Formulation: The paper introduces the concept of using experience-derived skills as privileged information for mathematical reasoning, framing the problem as teacher hypothesis validation rather than default imitation.
2. Framework (SGSD): A multi-teacher, skill-conditioned self-distillation framework that infers teacher polarity from the alignment between teacher support and rollout outcomes, utilizing a robust gated objective to handle noisy or conflicting signals.
3. Empirical Validation: Demonstrated on multiple mathematical reasoning benchmarks (AIME24, AIME25, HMMT25), showing that SGSD consistently outperforms GRPO and remains competitive with answer-conditioned self-distillation (OPSD) despite operating under a weaker PI assumption (skills vs. ground-truth answers).

4. Experimental Results

Experiments were conducted on Qwen3 models (1.7B, 4B, 8B) trained on DAPO-Math-17K and evaluated on competition-style benchmarks.

• Performance: On Qwen3-1.7B, SGSD improved the average accuracy across AIME24, AIME25, and HMMT25 from 37.4% to 43.7%. This represents a 6.2% improvement over GRPO and a 1.7% improvement over OPSD (which uses trusted reference answers).
• Stability: Unlike OPSD, which showed degradation in later training stages, SGSD maintained stable improvements. This is attributed to the outcome-validation mechanism filtering out misleading skill signals.
• Ablations:
  • Removing polarity validation led to performance collapse, confirming that blindly following skills is harmful.
  • Using a single teacher was inferior to the multi-teacher pool, highlighting the value of diverse hypotheses.
  • Robust support estimation (masking, clipping, thresholding) was critical for stability.
  • Adding reference answers to the skill-conditioned teacher caused performance drops, suggesting that mixing trusted and untrusted PI creates inconsistent update directions.

5. Significance and Claims

The paper claims that SGSD demonstrates that experience-derived skills can provide useful dense supervision even when their direction is not pre-verified, provided their polarity is validated against on-policy evidence.

• Weaker Assumption: SGSD achieves competitive results with answer-conditioned methods (OPSD) while only assuming the existence of a verifier and a skill bank, rather than ground-truth reference solutions.
• Mechanism: The success relies on treating retrieved skills as hypotheses to be validated rather than instructions to be followed. The gated objective effectively extracts informative teacher-student disagreements while suppressing noise and extreme signals.
• Limitations: The authors note that the method is currently evaluated primarily on mathematical reasoning with automatic verification. Extensions to open-ended generation would require calibrated judges. Additionally, the skill bank construction relies on a simple cold-start and hierarchical merging, which may benefit from more adaptive strategies in future work.

In summary, SGSD offers a pathway to enhance LLM reasoning in settings where ground-truth answers are unavailable, leveraging the model's own evolving experience to generate and validate its own teaching signals.