Technical Summary: TurnOPD

Problem Statement

On-policy distillation (OPD) is a promising framework for training language agents, where a student model learns from a stronger teacher by matching the teacher's distribution on the student's own generated trajectories. However, applying vanilla OPD to long-horizon agent tasks (involving multi-turn interactions, tool use, and environmental shifts) reveals two critical inefficiencies that degrade training performance and resource utilization:

1. External Mismatch (Rollout Depth): Vanilla OPD typically uses fixed, full-horizon rollouts. The authors identify that the reverse-KL supervision signal is heavily skewed toward early turns. As training progresses and trajectories extend, the "tail" turns often provide weak, noisy, or non-informative KL signals. Continuing to compute full-length rollouts wastes wall-clock resources on these low-yield steps.
2. Internal Mismatch (Loss Allocation): Standard trajectory-level KL objectives normalize loss based on token counts. Since early turns often contain more tokens and higher initial KL divergence, they dominate the loss budget. Consequently, deeper decision turns—which are crucial for long-horizon success but contain fewer tokens and lower raw KL—receive insufficient supervision. The authors formalize this as a contamination-compression mechanism: as the student generates more context, the next-token distribution becomes increasingly "forced" by formatting or repetitive patterns, compressing the observable KL even if genuine policy disagreements persist. This leads to a scenario where deep turns are under-trained once initial behaviors align.

Methodology: TurnOPD

To address these mismatches, the authors propose TurnOPD, a turn-level budgeting strategy that dynamically manages both the rollout horizon and the loss allocation. TurnOPD consists of two adaptive controllers:

1. Adaptive Rollout-Depth Budgeting

This controller dynamically determines the optimal rollout length ($H_{ctrl}$) to avoid wasting compute on low-signal tail turns. It synthesizes two signals:

• Efficiency Proxy ($H_{eff}$): Calculates the centroid of the survivor-weighted reverse-KL distribution. It treats the KL signal as a distribution of distillation mass, shifting the target depth based on where meaningful correction signals are concentrated.
• Coverage Lower Bound ($H_{cov}$): Ensures the rollout is deep enough to cover a specific quantile (e.g., 80%) of successful trajectories. This prevents the controller from truncating rollouts too aggressively, ensuring the student still observes sufficient successful completions.
• Mechanism: The controller updates $H_{ctrl} = \max(H_{eff}, H_{cov})$ using periodic full-length probe rollouts to estimate statistics, while routine training steps use the capped horizon.

2. Progressive Turn-Normalized Loss Budgeting

This controller addresses the internal mismatch by gradually shifting the loss aggregation strategy from token-level to turn-balanced supervision.

• Linear Blending: Instead of a hard switch, the method linearly interpolates between the standard trajectory-level weight (proportional to token count, $q_{traj}$) and a uniform turn-level weight ($q_{turn}$).
• Progression: The blending coefficient $\alpha$ increases linearly with training progress. Initially ($\alpha \approx 0$), the loss follows token mass to ensure stability. As training advances, $\alpha$ increases, shifting weight toward deeper turns to ensure they receive adequate supervision for refining complex decision-making.

Key Contributions

1. Turn-Level Diagnosis: The paper provides a systematic analysis of OPD in long-horizon settings, revealing that supervision signals and optimization budgets are non-uniformly distributed across turns. It identifies the "contamination-compression" phenomenon where context-induced constraints artificially suppress observable KL in deep turns.
2. Formalization of Mismatches: The authors formalize the "external mismatch" (fixed depth vs. varying signal utility) and "internal mismatch" (token-weighted loss vs. turn-level importance) as the root causes of vanilla OPD's inefficiency.
3. TurnOPD Framework: The proposal of a dual-controller system that adapts rollout depth and loss normalization simultaneously, tailored specifically for the turn-conditioned nature of agent interactions.
4. Empirical Validation: Extensive experiments demonstrating that TurnOPD advances the accuracy–time frontier, achieving superior performance with significantly reduced training time compared to vanilla OPD and other baselines.

Experimental Results

The authors evaluated TurnOPD on three representative benchmarks: ALFWorld (embodied planning), WebShop (web navigation), and Multi-Hop Search (information retrieval). They used various student-teacher pairs (e.g., Qwen3-1.7B/4B students distilled from Qwen3-8B/9B teachers).

• Accuracy-Time Frontier: TurnOPD consistently achieved the best "Least-Time" accuracy (highest accuracy reached within the minimal wall-clock time required for any method to complete 100 steps).
  • On ALFWorld-1.7B, TurnOPD improved Same-Step Avg@4 from 83.0 (Vanilla OPD) to 86.3 while reducing wall-clock time from 4.42 hours to 1.93 hours (a 2.29× speedup).
  • On WebShop, it improved accuracy while reducing time from 1.57h to 1.24h.
  • On Multi-Hop Search, it achieved the best Least-Time accuracy, outperforming vanilla OPD under shared time budgets.
• Ablation Studies:
  • Adaptive Depth Alone: Reduced wall time significantly (by ~55%) but slightly degraded accuracy, confirming that shortening rollouts saves compute but does not solve the loss-allocation problem.
  • Linear Blend Norm Alone: Improved accuracy by reallocating loss to deep turns but did not reduce wall time.
  • Combined: The full TurnOPD approach achieved the optimal trade-off, leveraging both efficiency and optimization levers.
• Controller Dynamics: Analysis showed that the adaptive controller successfully tracked the diagnostic signals, increasing rollout depth as successful trajectories emerged and balancing efficiency with coverage constraints.

Significance and Claims

The paper claims that TurnOPD offers a more efficient and effective framework for training long-horizon language agents. Its primary significance lies in shifting the paradigm of supervision from a flat token position to a turn-conditioned decision within an evolving interaction trace.

The authors assert that:

• Standard KL supervision and normalization along full trajectories are insufficient for long-horizon agents due to the specific structural mismatches identified.
• TurnOPD advances the accuracy–time frontier, enabling agents to reach higher performance levels with up to 2.29× faster training.
• The approach is robust across different model sizes and task types, suggesting that turn-aware budgeting is a generalizable principle for agent training.

The paper concludes that TurnOPD opens a path toward turn-aware and budget-adaptive training, addressing the specific challenges of long-horizon agent tasks that vanilla sequence-generation objectives fail to resolve.