🤖 AI

RTPO: Reverse-Turn Policy Optimization for Stabilizing Agentic RL Training

This paper introduces Reverse-Turn Policy Optimization (RTPO), a unified framework that stabilizes multi-turn agentic RL training by restructuring rollouts as sparse reverse trees to eliminate context mismatch, enable causally consistent credit assignment, and control asynchronous policy drift, thereby significantly outperforming existing baselines.

Original authors: Yugu Li, Jimmy Cao, Jianglin Qiao, Siyi Hu

Published 2026-08-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yugu Li, Jimmy Cao, Jianglin Qiao, Siyi Hu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving world of artificial intelligence, researchers are teaching large language models to act as autonomous agents. These are not just chatbots that answer questions; they are systems designed to plan, reason, and use external tools like calculators or web search engines to solve complex problems. To make these agents smarter, scientists often use a method called reinforcement learning. Imagine a student learning to play a new game: they try different moves, and if they win, they get a reward. Over time, they learn which moves lead to success. In the digital realm, the "moves" are the words the model generates, and the "reward" is a score given at the very end of a task. This approach has been highly successful for single-step tasks, such as solving a math problem in one go. However, when these agents are asked to perform long, multi-step workflows—like researching a topic, gathering data, and writing a report over several turns of conversation—the training process becomes notoriously unstable. The models often learn to solve short tasks but fail miserably when the task gets longer, their performance crumbling as the number of steps increases.

A team of researchers from universities in Australia and the United States has identified why this breakdown happens and proposed a new way to fix it. They discovered that the instability stems from three deeply connected problems that occur when training these agents over many turns. First, there is a mismatch between what the model sees while it is practicing and what it sees while it is being taught. During practice, the model might only see a summary of the conversation to save space, but during training, the computer forces it to look at the entire, full history of the chat. This difference confuses the model, making it unsure of which context to trust. Second, the system struggles to figure out which specific step in a long chain of reasoning actually led to the final success or failure. When a model gets a reward at the end of a ten-step process, the training algorithm often spreads that credit (or blame) evenly across all ten steps, even if only the fifth step was the critical mistake. This makes it hard for the agent to learn the right behavior. Third, as the training progresses, the model changes. If the system tries to learn from a long conversation that started when the model was "younger" and less skilled, but finishes training when the model is "older" and more skilled, the data becomes inconsistent. The model is essentially trying to learn from its own past self, creating a drift that destabilizes the learning process.

To solve these issues, the researchers developed a new framework called Reverse-Turn Policy Optimization, or RTPO. Instead of treating a long conversation as one giant, flat block of text, they reimagined the training process as a series of distinct, connected moments. The core idea is to train the model by working backward, starting from the very last step of a task and moving toward the beginning. When the system reaches a specific turn in the conversation, it pauses and creates several "sibling" versions of the future. It asks the model to imagine: "If I make this specific decision right now, what happens next?" It then runs these future scenarios forward to see if they lead to a good outcome. By comparing these sibling futures, the system can pinpoint exactly which decision at the current turn was the best one, without the noise of what happened before or the uncertainty of what might happen later. This method ensures that the model is always learning from a consistent context and that the credit for a successful outcome is assigned precisely to the decision that caused it.

The results of this new approach were striking. When tested on difficult mathematical reasoning tasks and complex web-search challenges, the new method significantly outperformed existing techniques. On a set of hard math problems, the new method improved the accuracy of the agents by more than twenty-one percent compared to the previous standard methods. On knowledge-based tasks, it improved performance by nearly eleven percent. Perhaps more importantly, the training process itself became much more stable. In previous methods, as the tasks got longer, the models would often get worse, but with this new approach, the models continued to improve even as the number of steps increased. The researchers also found that the agents learned to use tools more efficiently. On math problems, the agents made fewer tool calls but got more answers right, suggesting they learned to rely on their own reasoning and only use tools when absolutely necessary. On research tasks, they made more calls, but these calls were more strategic, leading to better information gathering.

The researchers verified that their solution worked by isolating each of the three problems they identified. They showed that by fixing the context mismatch, the model stopped getting confused about what it was supposed to be looking at. By fixing the credit assignment, the model learned to make better decisions at each specific step rather than guessing based on the final result. And by keeping the training data consistent with the current version of the model, they eliminated the drift that usually causes long training sessions to fail. The study provides a clear path forward for building more reliable AI agents that can handle complex, multi-step tasks without losing their way. It suggests that the key to mastering long-horizon reasoning is not just giving the model more data, but restructuring how it learns from that data, ensuring that every decision is judged fairly and accurately against the right future possibilities. This work marks a significant step toward creating AI assistants that can truly navigate the complexities of the real world, one thoughtful step at a time.

Technical Summary: Reverse-Turn Policy Optimization (RTPO) for Stabilizing Agentic RL Training

1. Problem Statement

Training multi-turn agentic workflows using Reinforcement Learning (RL) enables Large Language Models (LLMs) to perform complex reasoning, utilize external tools, and conduct iterative search. However, existing methods suffer from severe instability as the number of turns increases, often leading to performance degradation. The paper identifies three tightly coupled sources of this instability, which share a common structural origin in flattened trajectory optimization:

Rollout–Training Context Mismatch: In multi-turn settings, rollouts are often generated under truncated or summarized contexts (to manage context length), while training recomputes likelihood ratios under the full, concatenated history. This discrepancy biases the importance-sampling (IS) ratios used in policy optimization (e.g., PPO, GRPO), breaking the consistency between the generated and optimized turn-level contexts.
Weak Turn-Level Credit Assignment: Flattened methods assign a single trajectory-level advantage to all tokens. This obscures the specific contribution of individual turns, especially when terminal rewards are sparse. The trajectory advantage entangles the true turn-level signal with upstream state effects and downstream stochasticity, potentially reversing the gradient direction for specific turns.
Asynchronous Policy Drift: In asynchronous training, shorter trajectories may update the policy while longer trajectories are still being generated under an older policy. When these longer trajectories are eventually optimized, they are off-policy relative to the current model. Standard token-level clipping in PPO/GRPO cannot faithfully correct this drift over long horizons because the product of clipped ratios does not equal the true trajectory-level IS ratio.

2. Methodology: Reverse-Turn Policy Optimization (RTPO)

RTPO addresses these issues through a unified reverse-turn formulation that organizes multi-turn interactions as sparse trees and performs turn-level policy updates in temporal reverse order.

Core Components:

Turn-Boundary MDP: The interaction is modeled as a Hierarchical MDP where the state at turn $k$ is the interaction history $S_k$ . The policy is factorized into $K$ turn-level sub-policies $\pi_{\theta, k}$ , each mapping a conditioning context $c_k$ (which matches the rollout context exactly) to a complete response $l_k$ .
Reverse-Order Training: Training proceeds from the final turn ( $K-1$ ) back to the first turn ($0$). When optimizing turn $k$ , the downstream policies ( $k+1 \dots K-1$ ) are frozen. This ensures that the continuation value used for credit assignment is stationary and consistent with the current optimization target.
Sparse Tree Rollouts & State-Matched Siblings: For each turn $k$ , RTPO forks $G-1$ sibling rollouts from the shared boundary state $S_k$ . These siblings sample different turn- $k$ actions but share the same history. This allows for the estimation of turn-level advantages ( $A^H_{j,k}$ ) that are causally consistent, removing upstream state contamination and ensuring comparisons are made under matched downstream conditions.
On-Policy Continuation: To eliminate asynchronous drift, RTPO regenerates sibling continuations on-policy at each reverse step. When optimizing turn $k$ , the downstream turns are sampled using the current updated policy $\pi_{\theta > k}$ . This ensures the trajectory-level importance weight is identically one ( $\omega \equiv 1$ ), removing the need for unstable long-horizon IS corrections.

Theoretical Guarantees:

The paper provides three key theorems:

Convergence to Recursive Optimality: Under standard assumptions (finite state/action spaces, Robbins-Monro step sizes), reverse-order optimization with frozen downstream policies converges to a recursively optimal policy sequence.
Causally Consistent Advantage Estimation: The state-matched sibling estimator removes upstream state contamination terms present in trajectory-level baselines, reducing variance and bias in credit assignment.
Drift-Free On-Policy Continuation: By regenerating continuations on-policy, RTPO avoids the exponential variance growth associated with long-horizon importance sampling corrections.

3. Key Contributions

Identification of Instability Sources: The paper formally identifies and analyzes the coupled sources of instability in multi-turn agentic RL: context mismatch, weak credit assignment, and asynchronous drift.
Structural Diagnosis: It demonstrates that these issues share a common origin in flattened trajectory optimization and that existing methods (PPO, GRPO, Tree-based methods) do not jointly address this root cause.
RTPO Framework: Proposes a novel framework integrating rollout-training consistency, turn-level credit assignment, and on-policy continuation with theoretical guarantees on recursive optimality and reduced credit bias.
Empirical Validation: Validates RTPO on comprehensive multi-turn agentic RL benchmarks involving mathematical reasoning and knowledge-intensive search.

4. Experimental Results

Experiments were conducted using Qwen3-8B as the backbone across 12 benchmarks, including mathematical reasoning (GSM8K, MATH500, AIME24/25, AMC23, OE-Math) and knowledge reasoning (HotpotQA, 2Wiki, GAIA, WebWalkerQA, XBench, HLE).

Performance Gains: RTPO outperformed trajectory-level baselines (GRPO) by 21.50% and turn-level baselines (SeeUPO) by 10.76% in overall accuracy.
Stability: RTPO maintained stable rollout-training consistency (log-probability ratios near 1.0) throughout training, whereas baselines exhibited significant drift and recovery patterns.
Tool Usage Efficiency:
- In mathematical reasoning, RTPO achieved higher accuracy with fewer tool calls, indicating it learned to rely on internal reasoning and use tools only for precise verification, unlike baselines that used tools for repeated, inefficient verification.
- In knowledge reasoning, RTPO utilized more tool calls effectively to integrate external evidence, demonstrating that its optimization objective adapts tool-use strategies to the specific nature of the task rather than suppressing or encouraging calls indiscriminately.
Ablation Studies:
- Credit Assignment: Controlling for context mismatch, RTPO's turn-level credit assignment alone improved performance significantly over baselines.
- On-Policy Continuation: On long-horizon deep-search tasks, the on-policy variant of RTPO significantly outperformed an off-policy variant that reused stale continuations, confirming the theoretical benefits of avoiding policy drift.

5. Significance and Claims

The paper claims that RTPO provides a principled framework to stabilize multi-turn agentic RL training by structurally eliminating the mismatch between rollout and training contexts and controlling policy drift.

Theoretical Contribution: It establishes that reverse-order turn-level optimization, combined with state-matched sibling comparisons and on-policy continuation, can achieve recursive optimality and causally consistent credit assignment, addressing limitations inherent in flattened trajectory approaches.
Practical Impact: The method enables more stable training for tool-using agents, particularly in long-horizon scenarios where standard methods fail. It allows agents to learn efficient tool-use strategies tailored to the task (e.g., precise verification for math, thorough retrieval for knowledge) without suffering from the instability caused by context mismatches and asynchronous updates.
Limitations: The authors acknowledge that RTPO incurs higher training costs (approximately 41% more GPU hours) due to the need for sibling generation and reverse-order synchronization. They also note that the strict convergence guarantees apply to the tabular formulation, and the neural implementation relies on universal approximation assumptions. Additionally, the method's performance can degrade in late-stage training on small, sparse-reward datasets due to over-optimization.

In summary, RTPO offers a unified solution to the structural instabilities of multi-turn agentic RL, demonstrating that addressing the root causes of context mismatch and credit assignment through a reverse-turn formulation leads to superior performance and stability.