Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correction

Technical Summary: Missing Old Logits in Asynchronous Agentic RL

Problem Definition
In asynchronous reinforcement learning (RL) for Large Language Models (LLMs), the decoupling of sample generation (rollout) from policy optimization is essential for maximizing throughput. However, this architecture introduces a critical failure mode for Proximal Policy Optimization (PPO)-style off-policy correction.

Ideally, the importance ratio in PPO should be decomposed into two semantically distinct factors:

1. Training–Inference Discrepancy ($r_d$): Corrects for numerical differences between the inference engine (e.g., vLLM) and the training engine (e.g., Megatron-LM) at the same policy version.
2. Policy Staleness ($r_s$): Corrects for the gap between the historical behavior policy that generated the tokens and the current policy being optimized.

The paper identifies that in practical asynchronous pipelines, particularly those involving partial rollouts and delayed updates, the old training-side logits ($\pi_{old}$) required to compute the discrepancy term $r_d$ are often missing. When the actor updates its weights before the rollout corresponding to an earlier version is fully processed, the specific policy version $\pi_{old}$ used to generate those tokens may no longer be available.

This "missing-old-logit" problem forces existing methods to substitute the true $\pi_{old}$ with approximations (e.g., linear interpolation or static proxies). The authors argue that this substitution entangles discrepancy repair with staleness correction, breaking the intended semantic separation. Consequently, clipping and masking thresholds interact undesirably, leading to suboptimal optimization stability and performance.

Methodology
The paper proposes a unified analysis of off-policy corrections under a "dual-constraint" view and evaluates two primary directions for resolution:

1. Exact Old-Logit Acquisition (System-Level):
   The authors investigate three strategies to recover the ground-truth $\pi_{old}$ without relying on approximations:

   • Snapshot-based Version Tracking: Retaining historical parameter snapshots and reloading the specific version that generated each token. This is exact but incurs high memory and I/O overhead.
   • Dedicated Old-Logit Model: Maintaining a separate model instance solely to compute old logits while the main actor trains. This allows for computation overlap but requires resource partitioning.
   • Synchronization via Partial Rollout Interruption: Interrupting rollout workers to compute old logits for partial trajectories before the policy version is discarded. This avoids storage costs but introduces synchronization stalls and resource reconfiguration overhead.

2. Approximate Correction (Algorithm-Level):
   Recognizing that exact recovery is often too expensive for large-scale systems, the paper proposes a revised PPO-EWMA (Exponentially Weighted Moving Average) method.

   • Instead of using a static interpolation proxy, this method maintains a smoothed reference policy ($\pi_{prox}$) that tracks the center of the asynchronous version window.
   • Staleness-Aware Decay: The decay factor $\beta$ is set based on the expected staleness window ($W_{stale}$) to ensure the reference policy aligns with the middle of the version gap, rather than lagging behind.
   • Automatic Reset: To prevent the reference policy from drifting too far and causing the "Train-Infer Mask" (which filters tokens with high discrepancy) to collapse, the system monitors the fraction of active tokens. If this fraction drops below a threshold ($\tau$), the reference policy is reset to the current actor parameters.

The paper also provides a theoretical analysis (Proposition 1) demonstrating that interpolation-based proxies (linear or log-linear) do not genuinely recover the missing reference policy; rather, they merely re-parameterize the effective clipping boundaries of the total importance ratio.

Key Contributions

• Identification of the Missing-Old-Logit Problem: The paper formally defines the semantic failure mode in asynchronous Agentic RL where the absence of $\pi_{old}$ breaks the decoupling of discrepancy repair and staleness correction.
• Unified Analysis of Off-Policy Corrections: It formulates existing PPO variants under a Masked Importance Sampling (MIS) framework, clarifying why a single ratio or threshold is insufficient for handling both numerical discrepancies and policy staleness.
• System Trade-off Analysis: It evaluates three exact acquisition strategies, quantifying their costs in terms of memory, I/O, and synchronization overhead.
• PPO-EWMA Proposal: It introduces a low-cost, approximate correction method that preserves the benefits of decoupled correction without requiring expensive infrastructure changes.

Experimental Results
The authors evaluated their methods on Agentic RL benchmarks ($\tau^2$-Bench and VitaBench) using both dense (Qwen3-4B) and Mixture-of-Experts (Qwen3-30B-A3B) models.

• Performance: The PPO-EWMA method consistently outperformed standard Decoupled PPO and linear interpolation proxies (Linear_prox). On the 30B MoE model, PPO-EWMA achieved the highest pass rates on the airline split and tied for the best on retail and telecom tasks.
• Comparison to Ideal: While PPO-EWMA did not fully match the performance of the "Snapshot" method (which assumes perfect access to old logits), it achieved results very close to the ideal case, suggesting that a well-tuned approximate reference can recover most of the benefits of exact correction.
• System Overhead: Exact recovery methods (Snapshot) incurred significant system costs (e.g., ~178 seconds extra per step for the 30B model due to historical forward passes). In contrast, PPO-EWMA added negligible overhead, offering a superior performance-to-cost trade-off.
• Ablation Studies: Experiments confirmed that without the automatic reset mechanism, the EWMA reference could drift, causing the Train-Infer Mask to collapse and training to fail. The reset mechanism successfully re-centered the reference, maintaining stability.

Significance and Claims
The paper claims that the missing-old-logit problem is a fundamental semantic issue in asynchronous LLM RL that undermines the theoretical guarantees of decoupled correction. The authors assert that:

1. Exact recovery is the only way to achieve semantically correct discrepancy repair and staleness correction, but it is often prohibitively expensive.
2. Approximate methods based on simple interpolation are insufficient because they fail to restore the true reference policy, merely shifting constraint boundaries.
3. PPO-EWMA offers a practical, low-cost alternative that significantly improves training speed and optimization performance over existing asynchronous baselines, making it a viable solution for large-scale Agentic RL systems where exact logit recovery is infeasible.

The work concludes that while PPO-EWMA is an approximation and may struggle under extreme non-stationarity or very large version gaps, it represents a significant step forward in stabilizing asynchronous training without requiring heavy infrastructure modifications.