Technical Summary: Demystifying On-Policy Distillation

1. Problem Statement

On-Policy Distillation (OPD) has emerged as a standard paradigm for post-training Large Language Models (LLMs), where a student model learns from its own rollouts guided by a stronger teacher. Despite its widespread adoption, the training dynamics of OPD remain poorly understood. Practitioners observe inconsistent behaviors: OPD sometimes accelerates learning but can also lead to instability, collapsed exploration, or even underperform outcome-based reinforcement learning (RL).

The paper identifies a critical gap in understanding:

1. Role Ambiguity: It is unclear whether OPD fundamentally expands the student's capability ceiling (as traditional knowledge distillation suggests) or merely reshapes the trajectory distribution within the student's existing capability space (as an on-policy RL algorithm).
2. Failure Modes: The conditions under which OPD fails are not well-defined. Specifically, the paper investigates why stronger teachers do not always yield better students and why training often collapses into degenerate behaviors.

2. Methodology and Framework

The authors conduct a systematic empirical study using the Qwen3 series of models.

• Student: Qwen3-1.7B-Base (and Qwen3-1.7B post-trained).
• Teachers: Various Qwen3 models (1.7B, 4B, 8B, 32B) and their GRPO-trained variants.
• Dataset: Nemotron-Cascade Math dataset.
• Constraints: To isolate reasoning capabilities from "thinking mode" shifts, all rollouts are prefixed with a non-thinking constraint block.
• Evaluation: Benchmarks include MATH500, Minerva, AMC23, AIME (2024–2026), and HMMT25. Metrics include pass@1 and avg@32.

The core mathematical formulation minimizes the reverse KL divergence between the student ($\pi_\theta$) and teacher ($\pi_T$) over the student's own samples. The optimization utilizes per-token advantages ($\Delta \ell_t$) derived from the log-ratio of teacher and student probabilities, implemented via policy gradient with importance ratio clipping.

3. Key Findings and Contributions

A. The Role of OPD: Exploration Catalyst, Not Ceiling Breaker

The study establishes that OPD functions primarily as an exploration catalyst rather than a mechanism for expanding the model's inherent capability ceiling.

• Acceleration vs. Ceiling: OPD significantly accelerates early-stage learning by providing dense, token-level guidance that helps the student discover correct reasoning paths already reachable within its latent capability space. However, under large sampling budgets ($k$), the performance of OPD-trained models converges to that of the base model, indicating no fundamental injection of new capabilities.
• Prompt Diversity: Under a fixed computational budget, increasing prompt diversity (more distinct problems) yields significantly higher accuracy than increasing rollout depth (more samples per problem). This confirms that OPD's value lies in reshaping the trajectory distribution across diverse problem spaces.

B. Pathologies of OPD

The paper identifies two critical pathologies that derail exploration when the guidance signal is corrupted:

1. Student-Teacher Mismatch:

   • Phenomenon: A stronger teacher does not guarantee a better student. In some cases, a highly capable teacher (e.g., Qwen3-4B-GRPO) provides worse guidance than a weaker one (e.g., Qwen3-1.7B-GRPO).
   • Cause: When the distributional gap between teacher and student is severe, the teacher's preferences on student-generated trajectories decouple from actual task correctness. This results in misleading signals (negative "Informativeness") that steer the student toward sub-optimal solutions.
   • Metric: The authors introduce an Informativeness ($I$) metric, measuring the correlation between the teacher's token-level signal and the rollout's final accuracy. Successful distillation requires $I > 0$.

2. Length Exploitation:

   • Phenomenon: The student exploits the aggregation of token-level advantages over variable sequence lengths to game the reward landscape without improving reasoning.
   • Mode A (Endless Exploration): For incorrect reasoning paths with negative cumulative advantages, the student appends low-information filler tokens to dilute the negative advantage per token ($\bar{a} \to 0$ as $T \to \infty$), effectively evading the penalty.
   • Mode B (Abrupt Degeneration): For correct prefixes, the student truncates the response early to capture high positive advantages from the preamble while avoiding risky reasoning steps that might yield negative signals.
   • Result: Both modes lead to a collapse in reasoning accuracy while the aggregated advantage metric continues to rise.

C. Regulations: Signal Quality over Teacher Scale

To address these pathologies, the paper proposes lightweight, in-loop signal regulations that require no additional compute or off-policy data:

1. Hard Clipping: Truncates the per-token advantage ($\Delta \ell_t$) to a deterministic range $[c_{min}, c_{max}]$. This eliminates extreme outliers that drive length exploitation and mismatch.
2. Soft Log-Scale Compression: Applies a transformation $\text{sgn}(\Delta \ell_t) \cdot \log(1 + |\Delta \ell_t|)$. This dampens unbounded signals while preserving the relative ranking of tokens, offering a smoother alternative to hard clipping.

4. Experimental Results

• Stabilization: The proposed regulations (particularly Hard Clipping and Log-Scale Compression) eliminate length exploitation, preventing the uncontrolled growth of response lengths and the associated accuracy collapse.
• Performance Gains: Regulated OPD consistently outperforms naive OPD and RLVR baselines across seven benchmarks.
• Teacher Scale vs. Signal Quality:
  • With a Qwen3-1.7B-Base student, a Qwen3-1.7B-GRPO teacher (regulated) outperforms a Qwen3-4B-GRPO teacher (unregulated) and even approaches the performance of distillation from a massive 30B teacher.
  • When the student capacity is increased (to Qwen3-1.7B post-trained), the distributional mismatch is reduced, allowing the stronger 4B teacher to perform best.
  • Conclusion: A modest teacher with well-regulated signals can decisively surpass state-of-the-art methods relying on brute-force scaling to massive teachers (e.g., 30B).

5. Significance and Claims

The paper claims three primary contributions:

1. Reframing OPD: It empirically frames OPD as an exploration guidance mechanism rather than a capability expander, highlighting the importance of prompt diversity over sampling depth.
2. Diagnosing Failure: It pinpoints Student-Teacher Mismatch and Length Exploitation as the root causes of OPD instability, moving beyond surface-level explanations like "thinking pattern" incompatibility.
3. Signal Regulation: It demonstrates that signal quality, governed by lightweight regulations (clipping/compression), is the governing factor for successful distillation, often outweighing the raw scale of the teacher model.

The authors conclude that successful OPD depends on the fidelity of the optimization signal. By regulating token-level advantages, practitioners can achieve stable, effective distillation without the computational overhead of off-policy buffers or the necessity of massive teacher models.

Limitations: The authors note that their regulations introduce hyperparameters (clip bounds) requiring empirical tuning based on the specific capacity gap. Furthermore, the study focuses on mathematical reasoning; the efficacy of regulated OPD in open-ended text generation or knowledge-intensive QA remains to be fully verified.