Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalization

1. Problem Statement

The paper addresses the critical instability observed when applying Reinforcement Learning (RL) algorithms, specifically Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO), to multi-turn LLM agent training under off-policy conditions.

While RL has improved LLM reasoning and tool use, practical training pipelines often require reusing data batches (mini-batches) due to computational constraints. This creates a hybrid update mechanism where later updates become increasingly off-policy (the data distribution drifts from the current policy). The authors identify two fundamental sources of instability in this setting:

1. Granularity Mismatch: Standard PPO performs token-level credit assignment, which misaligns with the natural structure of multi-turn interactions (e.g., distinct phases of problem analysis, query formulation, and information processing). This leads to noisy updates where individual tokens compete for credit rather than reinforcing the entire turn.
2. High-Variance Off-Policy Updates: As training progresses, the distribution mismatch increases the variance of importance sampling ratios. This results in unreliable gradient estimates, "reward hacking," and catastrophic performance collapse (policy collapse), even when standard clipping mechanisms are used.

2. Methodology: The SORL Framework

The authors propose SORL (Stabilizing Off-Policy Reinforcement Learning), a framework designed to align policy optimization with multi-turn structures and suppress unreliable updates. SORL introduces two core mechanisms:

A. Turn-Level Importance Sampling

Instead of treating every token as an independent optimization unit, SORL aggregates credit assignment at the turn level.

• Mechanism: The policy objective is reformulated to group tokens by their turn boundaries ($[t_{start}^k, t_{end}^k]$).
• Weighting: A turn-level importance weight ($w_{turn}^k$) is calculated as a length-normalized geometric mean of the token-level ratios within that turn.
• Effect: All tokens within a single turn share a common, aggregated advantage signal. This aligns the optimization granularity with the natural "action" structure of agents (a turn is an action), reducing variance amplification caused by token-level fluctuations.

B. Clipping-Triggered Normalization

To address the instability caused by highly off-policy batches, the authors introduce a dynamic normalization mechanism based on the clipping bias.

• Gradient Decomposition: The PPO gradient is decomposed into a standard policy gradient term and a clipping bias term ($C(\theta)$), which represents the contribution of tokens where clipping was active (i.e., the importance ratio exceeded the trust region).
• Observation: The norm of the clipping bias correlates strongly with the degree of off-policyness and gradient explosion. Large clipping norms indicate that the current batch is dominated by unreliable samples.
• Mechanism: The algorithm computes the $L_2$ norm of the clipping bias term for the current batch and uses its inverse to rescale the policy gradient:
  $$\nabla_\theta J_{SO}(\theta) \propto \frac{1}{\|C(\theta)\|^2} \nabla_\theta J_{Turn-PPO}(\theta)$$
• Effect: This adaptively downweights updates from batches with high off-policy variance, preventing gradient spikes and ensuring conservative learning dynamics without requiring early stopping.

Instantiations

The framework is algorithm-agnostic and is instantiated as:

• SO-PPO: Combines turn-level importance sampling with clipping-triggered normalization.
• SO-GRPO: Applies clipping-triggered normalization to GRPO (omitting turn-level sampling since GRPO already uses trajectory-level grouping).

3. Key Contributions

1. Empirical Diagnosis: The authors provide a rigorous analysis identifying that token-level optimization misalignment and off-policy variance accumulation are the root causes of PPO/GRPO collapse in multi-turn agent tasks.
2. Unified Stabilization Framework: They propose SORL, which integrates turn-level credit assignment (to fix granularity mismatch) and clipping-triggered normalization (to fix off-policy variance). This yields more conservative and robust gradient estimates.
3. Algorithm Agnosticism: The framework is demonstrated to be effective across different RL algorithms (PPO and GRPO), proving that the stabilization principles are generalizable beyond specific objective functions.
4. Elimination of Heuristics: The method achieves stability without relying on careful early stopping, manual checkpoint selection, or complex hyperparameter tuning.

4. Experimental Results

The authors evaluated SO-PPO and SO-GRPO on multi-turn search benchmarks (Natural Questions, HotpotQA) and medical multiple-choice QA tasks (MedQA, MedMCQA, etc.).

• Training Stability:
  • Standard PPO and GRPO exhibited sharp performance collapses and exploding gradient norms (up to $10^8$) during training.
  • SORL variants maintained stable optimization trajectories, kept gradient norms in a normal scale, and prevented performance collapse entirely.
  • SO-PPO and SO-GRPO consistently maintained lower clipping ratios, indicating more reliable updates.
• Task Performance:
  • Search Tasks: SORL achieved superior or comparable Exact Match (EM) scores compared to baselines like Search-R1, with the added benefit of not requiring early stopping.
  • Medical QA: On 8B-scale models, SO-GRPO achieved the highest average accuracy (55.34%) across in-domain and out-of-domain medical benchmarks, significantly outperforming standard RAG, Chain-of-Thought (CoT), and the Search-R1 baseline.
• Ablation Studies: The method remained stable even under extreme off-policy conditions (smaller mini-batches), confirming the effectiveness of the normalization mechanism.

5. Significance

This work provides a practical and scalable solution for training long-horizon LLM agents. By addressing the structural mismatch between token-level optimization and turn-level interactions, and by dynamically correcting for off-policy variance, SORL enables the reliable training of complex agentic systems.

The significance lies in:

• Reliability: It removes the "black box" nature of RL training collapse, allowing practitioners to train agents for longer horizons without fear of catastrophic failure.
• Efficiency: It improves sample efficiency by allowing the safe reuse of off-policy data, which is crucial given the high cost of generating multi-turn agent trajectories.
• Generalizability: The approach is not limited to PPO but offers a blueprint for stabilizing other RL algorithms in agentic settings, potentially accelerating the deployment of autonomous AI agents in critical domains like healthcare and scientific research.