Technical Summary: Multi-Rollout On-Policy Distillation via Peer Successes and Failures

Problem Statement

Large Language Models (LLMs) are frequently post-trained using reinforcement learning with verifiable rewards (RLVR). In this paradigm, sampled solutions are scored by answer checkers, unit tests, or task-specific verifiers. While effective, these rewards are typically sparse: they indicate whether an entire trajectory succeeds or fails but provide limited guidance regarding which specific tokens or reasoning steps caused the outcome.

On-Policy Distillation (OPD) addresses the distribution mismatch inherent in offline distillation by training the student on trajectories sampled from its own policy, converting teacher judgments into dense token-level signals. However, existing OPD methods suffer from a critical inefficiency: they treat each rollout independently. When multiple rollouts are generated for the same prompt, the teacher signal for one trajectory is computed without access to the other attempts sampled from the same student. This design discards the local structure of the rollout group. For reasoning-intensive tasks, this local structure is highly informative; successful rollouts reveal valid solution strategies, while failed rollouts expose plausible but incorrect reasoning paths, missing constraints, or execution errors. Treating rollouts in isolation prevents the teacher from comparing alternatives to identify where an unsuccessful trajectory diverges from a successful one.

Methodology: Multi-Rollout On-Policy Distillation (MOPD)

The authors propose Multi-Rollout On-Policy Distillation (MOPD), a peer-conditioned distillation framework that explicitly utilizes the student's local rollout group to construct more informative teacher signals.

Core Mechanism

For each prompt $x$, the student samples a group of $N$ on-policy trajectories, $Y(x) = \{y^{(i)}\}_{i=1}^N$. These are scored by a verifier $R$ and partitioned into successful ($Y^+$) and failed ($Y^-$) sets based on a threshold $\tau$.

When supervising a target rollout $y^{(i)}$, MOPD conditions the teacher distribution $q_\phi$ not only on the prompt and the target's prefix but also on peer rollouts from the same group (excluding the target itself). The teacher distribution becomes:
$$q_{\text{peer}}^\phi(\cdot \mid x, y^{(i)}_{<t}) = q_\phi(\cdot \mid x, C_i(x), y^{(i)}_{<t})$$
where $C_i(x)$ is the peer context constructed from $Y^+_{-i}(x)$ and $Y^-_{-i}(x)$.

Peer-Context Construction Strategies

The paper investigates two primary strategies for constructing $C_i(x)$:

1. Positive Peer Imitation: The teacher is conditioned only on successful peer trajectories. This treats successful rollouts as "peer experts," providing positive evidence for valid reasoning patterns and anchoring the student to multiple valid derivation styles rather than a single reference.
2. Contrastive Success–Failure Conditioning: The teacher is conditioned on both successful and failed peer trajectories. Successful peers indicate what the target should resemble, while failed peers serve as structured negative evidence, identifying misleading solution paths to avoid. This contrastive approach helps the teacher distinguish between superficially similar correct and incorrect trajectories, particularly in tasks with sparse terminal rewards.

The training objective minimizes the divergence (Reverse KL for math, Jensen-Shannon for others) between the student's distribution and the peer-conditioned teacher distribution across the token sequence.

Key Contributions

1. Framework Innovation: MOPD is the first framework to explicitly condition the teacher on the local trial-and-error space of a student's multi-rollout group, transforming independent trajectory supervision into a comparative learning process.
2. Dual Evidence Utilization: The method leverages both successful peers (positive evidence) and failed peers (negative evidence). The authors demonstrate that failed peers are not merely noise but provide structured signals about plausible mistakes, constraints, and formatting errors.
3. Signal Quality Analysis: The paper introduces a direct analysis of self-teacher signal quality, measuring how well different context constructions align teacher scores with ground-truth verifier rewards. This avoids confounds associated with simple answer-recovery tests.
4. Empirical Validation: Extensive experiments across competitive programming, mathematical reasoning, scientific QA, and tool use demonstrate consistent improvements over standard on-policy baselines (GRPO, SDPO).

Experimental Results

The authors evaluated MOPD on four families of tasks:

• Mathematical Reasoning (AIME2024/2025, HMMT25): MOPD substantially improved mean@8 compared to SDPO. Notably, SDPO, which relies on rich environment feedback, underperformed on math tasks where only binary verifier rewards are available, whereas MOPD's peer-augmented supervision proved highly effective.
• Competitive Programming (LiveCodeBench v6): MOPD achieved state-of-the-art results among the compared methods, reaching 61.82 mean@8 and 67.23 pass@8 on Qwen3-8B, outperforming SDPO and GRPO.
• Science QA and Tool Use: MOPD achieved the best performance across all sub-domains (Biology, Chemistry, Physics, Materials) and tool-use benchmarks.

Key Findings from Analysis

• Mixed Context Superiority: The "2 Success + 1 Failure" context consistently achieved the highest alignment with verifier rewards across ranking (Spearman, Kendall's $\tau$) and discrimination (Pairwise Accuracy, Success AUC) metrics.
• Role of Failures: While successful peers are the primary driver of signal quality, adding a failed peer to a success-only context significantly sharpens decision boundaries (e.g., increasing Pairwise Accuracy from 0.32 to 0.40).
• Diversity: MOPD generates solutions with significantly higher lexical (Distinct-1/2) and structural (AST Jaccard distance) diversity compared to baselines, suggesting that peer conditioning encourages exploration of varied solution strategies.
• Efficiency: The computational overhead of MOPD is modest, with total per-step time increasing by only 9.1% on LiveCodeBench and 33.3% on Science QA compared to SDPO.

Significance and Claims

The paper argues that effective on-policy distillation should exploit the student's multi-rollout trial-and-error behavior rather than treating rollouts as isolated samples. By conditioning the teacher on peer successes and failures, MOPD turns the teacher into a local diagnostic model capable of recognizing instance-specific errors and distinguishing plausible failures from correct solutions.

The authors claim that the gains arise from more faithful, instance-adaptive supervision. The mixed success–failure context better aligns teacher scores with verifier rewards, indicating that the method successfully leverages the structured information available within a single sampling group. This approach bridges the gap between reinforcement learning (which uses group-relative advantages) and distillation (which uses token-level supervision), offering a pathway to improve reasoning capabilities in smaller models without the need for external, larger teachers or complex reward modeling.

The work concludes that while peer conditioning introduces additional context length, the resulting improvements in correctness, diversity, and training stability justify the cost, particularly in domains where verifiable rewards are available but sparse.