SG-OPD: Sign-Gated On-Policy Distillation via Sign-Consistency Gating and Phased Teacher Sampling

Technical Summary: Sign-Gated On-Policy Distillation (SG-OPD)

1. Problem Statement

The paper addresses limitations in On-Policy Distillation (OPD), a technique where a student model is trained on its own trajectories using dense per-token supervision from a stronger teacher model. While OPD often outperforms off-policy distillation and standard reinforcement learning (RL), the authors identify that its effectiveness relies on two implicit assumptions that frequently fail in practice, particularly in strong-to-weak settings (e.g., distilling a large reasoning model into a smaller one):

1. Fragile Trajectory-Level Alignment: In cold-start scenarios, a weak student's rollouts often lie far from the teacher's trajectory support. Consequently, the reverse-KL signal (the core of OPD) is dominated by noise rather than informative supervision, leading to unstable early training.
2. Non-Uniform Token-Level Reliability: Even on rollouts verified as correct by an external verifier, the teacher's per-token preferences may contradict the "verifier-correct" direction. The teacher might assign low probability to tokens that are actually necessary for a valid solution path. Blindly amplifying the reverse-KL gradient on these tokens penalizes valid trajectories.

2. Methodology: SG-OPD

The authors propose Sign-Gated On-Policy Distillation (SG-OPD), a framework that utilizes a binary verifier as a trust signal for the teacher at two complementary granularities: sample level and token level.

A. Sample Level: Phased Teacher Sampling (PTS)
To address the trajectory-level alignment failure at cold-start, SG-OPD introduces an auxiliary loss based on verified teacher rollouts.

• Mechanism: During the early stages of training, the mini-batch is split. A fraction of prompts are sampled from the teacher ($\pi^*$), and only trajectories verified as correct are retained. These serve as an "anchor" via a Cross-Entropy (CE) loss.
• Annealing: This teacher guidance is weighted by a three-phase cosine schedule ($\alpha(t)$). It starts high during cold-start to bridge the distribution gap and decays to zero, ensuring the asymptotic training distribution remains fully on-policy (student-generated).

B. Token Level: Sign-Consistency Gating
To address the token-level reliability failure, SG-OPD routes updates based on the agreement between the verifier's outcome signal and the teacher's preference signal.

• Signals:
  • $a_1(t)$: The outcome-level advantage from a binary verifier (e.g., GRPO style), indicating if the trajectory is correct.
  • $a_2(t)$: The reverse-KL advantage from the teacher (OPD signal).
• The Gate: A binary gate $g_t$ is defined as $1[a_1(t) \cdot a_2(t) > 0]$.
  • Consensus Tokens ($g_t=1$): The verifier and teacher agree on the direction. These tokens undergo extrapolation (amplified update) using a higher extrapolation strength ($\lambda_{high}$).
  • Conflict Tokens ($g_t=0$): The signals disagree (e.g., the teacher suppresses a token that leads to a correct answer). These tokens undergo interpolation (softened update) or are masked, preventing the student from being pushed away from verified solutions.
• Stability: A detached clipping weight is applied to prevent large outlier advantages from dominating the gradient.

The final objective combines the token-level gated policy gradient loss with the phased sample-level teacher anchor loss.

3. Key Contributions

1. Identification of Failure Modes: The paper explicitly identifies and analyzes two structural limitations of OPD: the fragility of trajectory-level alignment at cold-start and the non-uniform reliability of teacher preferences even on correct trajectories.
2. SG-OPD Framework: The proposal of a two-granularity framework that treats a binary verifier as a trust signal. It combines Phased Teacher Sampling to stabilize cold-start and Sign-Consistency Gating to selectively extrapolate or interpolate token updates based on signal agreement.
3. Empirical Validation: Extensive experiments on competition-level mathematical reasoning benchmarks demonstrate that SG-OPD consistently outperforms standard OPD and extrapolated variants (ExOPD), particularly in regimes where uniform extrapolation causes training collapse.

4. Experimental Results

Experiments were conducted using Qwen3-1.7B as the student and Qwen3-4B-Non-Thinking-RL-Math (step 500) as the teacher, evaluated on four benchmarks: AIME24, AIME25, HMMT25-Feb, and HMMT25-Nov.

• Performance Gains: SG-OPD achieved the highest average scores across all metrics.
  • avg@32: 29.53% (an improvement of +1.98 over OPD and +1.54 over ExOPD).
  • pass@32: 59.17% (an improvement of +7.50 over OPD and +5.00 over ExOPD).
• Ablation Studies:
  • The Sign-Consistency Gate was found to widen the "safe range" for extrapolation strength. While uniform extrapolation (ExOPD) collapsed at high $\lambda$ (e.g., $\lambda=1.8$), SG-OPD maintained or improved performance at the same strength.
  • PTS alone improved performance over OPD, but the combination of PTS and the Sign-Gate yielded nearly additive gains, confirming their orthogonality.
• Training Dynamics: SG-OPD achieved higher training rewards and maintained higher policy entropy compared to OPD and ExOPD, indicating that the gating mechanism amplifies consensus without collapsing exploration.

5. Significance and Claims

The paper claims that SG-OPD provides a more robust and stable approach to on-policy distillation by explicitly handling the misalignment between student trajectories and teacher preferences.

• Trust Signal: It demonstrates that a binary verifier can serve as a reliable trust signal for the teacher, allowing the model to "back off" where the teacher is unreliable (conflict tokens) and "lean in" where they agree (consensus tokens).
• Stability: The method remains stable at extrapolation strengths that cause uniform extrapolation methods to diverge, effectively turning "untrainable" regimes into high-performing ones.
• Scope: The authors note that the current validation is limited to math reasoning with binary verifiers. They suggest the approach generalizes to other verifier-style tasks (e.g., code with unit tests) but acknowledge that process-level certificates (step-by-step verification) would be a necessary extension for tasks where the final answer is wrong but intermediate steps are useful.

The work is presented as a methodological improvement to the distillation pipeline, offering a solution to specific structural failure modes observed in strong-to-weak distillation settings.