Technical Summary: Uni-OPD

Problem Statement

On-policy distillation (OPD) has emerged as a promising post-training paradigm for consolidating the capabilities of specialized expert models into a single student model by combining on-policy sampling with token-level supervision. However, despite empirical successes, the conditions for reliable OPD improvement remain poorly understood. The authors identify two fundamental bottlenecks limiting effective OPD:

1. Insufficient Exploration: Students often fail to explore informative states, specifically diverse and appropriately difficult self-generated trajectories.
2. Unreliable Teacher Supervision: Token-level guidance from teachers can be unreliable for student rollouts. Specifically, aggregated token-level guidance may fail to remain order-consistent with the outcome reward (i.e., correct trajectories do not necessarily receive higher aggregated scores than incorrect ones). This misalignment can occur due to out-of-distribution (OOD) degradation, overestimation of incorrect trajectories, or underestimation of correct ones.

Existing OPD research is largely confined to Large Language Models (LLMs) or limited subsets of Multimodal Large Language Models (MLLMs), lacking a unified framework that generalizes across modalities and addresses these reliability issues.

Methodology: Uni-OPD

The authors propose Uni-OPD, a unified OPD framework designed for both LLMs and MLLMs. The framework employs a dual-perspective optimization strategy targeting the student's exploration and the teacher's supervision reliability.

1. Student Perspective: Data Balancing for Exploration

To address insufficient exploration, Uni-OPD introduces two complementary data-balancing strategies:

• Offline Difficulty-Aware Data Balancing: Instead of aggressively filtering out "easy" (always correct) or "hard" (always incorrect) samples—a practice common in RL that reduces diversity—the authors observe that training data often follows mirrored J-shaped or U-shaped difficulty distributions. Uni-OPD selectively upsamples mid-difficulty samples (those correct in only some of multiple rollouts). This reshapes the distribution to be more uniform, preserving a balanced spectrum of difficulty and ensuring the student is exposed to diverse solution paths.
• Online Correctness-Aware Data Balancing: To prevent the model from collapsing into local optima when rollout groups lack outcome diversity (e.g., all trajectories are incorrect), the method enforces a balanced composition of correct and incorrect trajectories within each rollout group during training. This ensures the student consistently receives meaningful contrastive signals.

2. Teacher Perspective: Outcome-Guided Margin Calibration

To address unreliable supervision, Uni-OPD introduces a mechanism to restore order consistency between token-level guidance and outcome rewards.

• Trajectory-Level Distillation Return: The method defines a trajectory-level return ($G_{OPD}$) as the average log-probability gap between the teacher and the student.
• Order Consistency Criterion: A reliable teacher should assign higher returns to correct trajectories ($S^+$) than to incorrect ones ($S^-$). The authors define a prompt-level margin $m(q)$ as the difference between the minimum return of correct trajectories and the maximum return of incorrect trajectories.
• Calibration Mechanisms: If the margin $m(q)$ falls below a safety threshold $\delta$, the system applies one of two strategies:
  • Margin Mask: Discards prompt groups where the ordering is violated (or greedily removes the most adversarial trajectories) to ensure training only occurs on reliable signals.
  • Margin Shift: Applies a minimal additive correction to the trajectory returns to restore the margin to $\delta$. Specifically, it shifts the returns of correct trajectories upward (or incorrect ones downward) to enforce the ordering constraint without discarding data.

Key Contributions

1. Identification of Bottlenecks: The paper identifies insufficient exploration of informative states and unreliable teacher supervision (specifically the lack of order consistency with outcome rewards) as the core limitations of current OPD.
2. Dual-Perspective Optimization Recipe: The authors propose a unified framework that jointly improves student exploration via offline/online data balancing and teacher supervision via outcome-guided margin calibration.
3. Comprehensive Validation: The method is validated across 5 domains and 16 benchmarks, covering:
   • Single-teacher and multi-teacher distillation.
   • Distillation across LLMs and MLLMs.
   • Strong-to-weak distillation (transferring capabilities from larger to smaller models).
   • Cross-modal distillation (unifying textual and multimodal capabilities).

Experimental Results

Extensive experiments on Qwen3 family models (LLMs and MLLMs) demonstrate the effectiveness of Uni-OPD:

• Performance Gains: Uni-OPD consistently outperforms standard OPD and other baselines (SFT, ExPO, ExOPD) across math reasoning, code generation, logic reasoning, and document understanding. For instance, in multi-teacher distillation on LLMs, Uni-OPD improved math reasoning scores by 1.5% and code generation by 3.4% over standard OPD.
• Strong-to-Weak Distillation: Uni-OPD effectively bridges the capacity gap, enabling smaller students (e.g., 1.7B and 4B) to absorb complex reasoning behaviors from larger teachers (e.g., 30B), outperforming standard OPD by significant margins.
• Cross-Modal Capabilities: In cross-modal settings, Uni-OPD successfully unifies capabilities from text-only and multimodal teachers into a single MLLM student, improving both textual code generation and multimodal math reasoning simultaneously.
• Efficiency: Uni-OPD achieves superior performance with substantially fewer optimization steps compared to Reinforcement Learning (RL) baselines, converging faster while maintaining high reasoning capabilities.
• Ablation Studies: Removing either the data balancing strategies or the margin calibration mechanism results in performance degradation, confirming that both student exploration and teacher reliability are critical drivers of success.

Significance and Claims

The paper claims that Uni-OPD provides a practical foundation for reliable and scalable distillation. Its significance lies in:

• Generalizability: It offers a unified recipe that works across different model sizes (strong-to-weak), modalities (text and vision), and teacher configurations (single and multi-teacher).
• Reliability: By explicitly addressing the order-consistency issue between token-level signals and outcome rewards, the method stabilizes training and prevents the degradation often seen in naive OPD.
• Modality-Agnostic Reasoning: The results suggest that reasoning capabilities distilled via OPD are largely independent of modality, allowing for the efficient merging of specialized experts into a single, versatile reasoner.

The authors conclude that their work provides practical insights into the mechanics of reliable OPD and opens avenues for future research in scalable capability merging and agentic planning.