Technical Summary: Weak-to-Strong Generalization via Direct On-Policy Distillation

1. Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR) has become a dominant paradigm for enhancing the reasoning capabilities of Large Language Models (LLMs). However, RLVR is computationally expensive because every training update requires the target model to generate rollouts, receive verifiable scores, and update its policy. As model scales increase, the cost of generating these rollouts grows, making the re-running of RLVR from scratch for every new, stronger model a significant bottleneck.

The paper addresses the challenge of Weak-to-Strong Generalization: Can a model trained via RL on a smaller, weaker "teacher" model be used to improve a larger, stronger "student" model?

• The Limitation of Naive Distillation: Simply distilling the final policy of a post-RL weak teacher (On-Policy Distillation or OPD) is insufficient. The teacher's final policy entangles the useful improvements learned via RL with the intrinsic capacity limits of the smaller model. If the student is already stronger than the teacher, imitating the teacher's final distribution can degrade the student's performance by overwriting its superior behaviors.
• The Core Question: How can one transfer the RL-induced improvement signal from a weak model to a strong model without inheriting the weak model's capacity ceiling?

2. Methodology: Direct On-Policy Distillation (Direct-OPD)

The authors propose Direct-OPD, a paradigm that transfers the policy shift induced by RL on a weak model, rather than the teacher's final policy itself.

2.1 The Core Signal: Policy Shift as Implicit Reward

Instead of training the student to match the teacher's output distribution $\pi_T$, Direct-OPD isolates the change caused by RL by contrasting the post-RL teacher $\pi_T$ with its pre-RL reference $\pi_{Tref}$.
The policy shift is defined as:
$$\Delta_T(y | x) = \log \pi_T(y | x) - \log \pi_{Tref}(y | x)$$
Under the framework of KL-regularized RL, this log-ratio is mathematically equivalent to the implicit reward signal that trained the teacher (up to a scaling factor and a constant). Thus, the pair of checkpoints $(\pi_{Tref}, \pi_T)$ encodes the reward signal directly in policy space.

2.2 The Objective Function

Direct-OPD applies this shift signal to the student's own on-policy states. Let $\pi_\theta$ be the student policy initialized from $\pi_S$. The objective optimizes the student to follow the teacher's shift direction while regularizing against drifting too far from its own initialization:
$$J_{Direct-OPD}(\theta) = \mathbb{E}_{x \sim D} \left[ \mathbb{E}_{y \sim \pi_\theta} [\Delta_T(y | x)] - \alpha D_{KL}(\pi_\theta(\cdot | x) \parallel \pi_S(\cdot | x)) \right]$$

• Dense Implicit Reward: The term $\Delta_T$ acts as a dense, per-token reward. For a token $v$ at state $s_t$, the immediate reward is $r_t(v) = \log \pi_T(v | s_t) - \log \pi_{Tref}(v | s_t)$.
• On-Policy Sampling: The student samples its own trajectories. The gradient is computed on the student's visited prefixes, ensuring the signal is applied to states the student actually encounters.
• Top-K Approximation: To make training efficient, the method restricts the reward calculation to the student's top-$k$ tokens at each step, using a Rao-Blackwellized estimator to reduce variance.

2.3 Adaptive KL Control

The magnitude of the implicit reward $\Delta_T$ depends on the teacher's training hyperparameters (specifically the KL budget $\beta$), which are unknown to the student. A fixed KL coefficient $\alpha$ may not scale robustly across different teacher-student pairs.
The authors introduce a lightweight adaptive KL controller that adjusts $\alpha$ based on the batch-level mean of the weighted shift signal. If the average shift is positive (indicating the teacher's RL generally increased probability on retained tokens), $\alpha$ increases to prevent over-amplification; if negative, $\alpha$ decreases to allow the student to move away from suppressed tokens.

3. Key Contributions

1. A New Weak-to-Strong Paradigm: The paper recasts small-model RL not as a source of a final policy to imitate, but as a cheap generator of an implicit reward signal. By evaluating the teacher's policy shift on the student's own states, Direct-OPD transfers the direction of improvement rather than the capacity of the teacher.
2. Consistent Gains Across Scales: The method consistently improves stronger students using weaker teachers. Notably, it improves students (e.g., R1-Distill-7B, Qwen3-4B) that already outperform the post-RL teacher, a scenario where standard distillation fails.
3. Compute Efficiency: Direct-OPD is significantly cheaper than running RL directly on large models. It leverages the stability and speed of small-model RL and adds only a short, inexpensive transfer phase.
4. Sequential Composition: The method allows for the sequential composition of multiple policy shifts from different teacher pairs, enabling the accumulation of distinct reasoning abilities in a single student.

4. Experimental Results

The authors evaluate Direct-OPD on mathematical reasoning benchmarks (AIME 2024 and AIME 2025) using various model families (Qwen3, R1-Distill, Nemotron).

• Weak-to-Strong Transfer:
  • Qwen3-1.7B: Improved from 48.3% to 58.3% on AIME 2024 in just 4 hours using 8 A100 GPUs, transferring the shift from a 1.5B teacher.
  • Stronger Students: Successfully improved R1-Distill-7B (initially 56.7%) and Qwen3-4B (initially 72.5%), both of which started with higher accuracy than the post-RL teacher (51.3%). Standard OPD degraded performance in these cases.
• Comparison with Direct RL:
  • Under a matched-step budget, running RL on a 1.5B model and transferring via Direct-OPD outperformed running RL directly on a 7B model in both accuracy and compute time.
  • The transfer stage adds negligible time (~4 hours) compared to the weeks required for direct large-scale RL.
• Sequential Composition:
  • Applying two distinct policy shifts (from JustRL and QuestA teachers) sequentially to Qwen3-1.7B resulted in cumulative gains, pushing AIME 2024 accuracy from 48.3% to 63.8%.
• Generalization Properties:
  • Token Overlap: Unlike standard OPD, Direct-OPD does not require high top-k token overlap between teacher and student. It works even when the student's thinking patterns differ significantly from the teacher's.
  • Horizon: Short-horizon training (2k tokens) generalizes effectively to long rollouts, capturing the transferable signal without the noise of very long sequences.

5. Significance and Claims

The paper claims that RL outcomes can be reused across model scales as implicit reward signals, not merely as final models to imitate.

• Reframing RL: The work shifts the perspective of RL training from producing a "final model" to producing a "policy shift" that encodes a reward landscape. This shift is a transferable asset that is independent of the specific model capacity that generated it.
• Scalability: By decoupling the expensive RL search (done on small models) from the final deployment (on large models), Direct-OPD offers a scalable path for post-training reasoning models without the prohibitive cost of re-running RL on every new, larger architecture.
• Robustness: The method is robust to mismatches in model families and initial capabilities, provided the RL-induced shift is meaningful on the student's visited states.

The authors conclude that Direct-OPD provides a practical, efficient, and theoretically grounded mechanism for weak-to-strong generalization, enabling the reuse of RL supervision signals across different model scales and architectures.