🤖 machine learning

Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning

This paper introduces Single-rollout Asynchronous Optimization (SAO), a stable and efficient asynchronous reinforcement learning framework that replaces group-wise sampling with single-rollout strategies and employs strict token-level clipping to successfully train large-scale agentic models like GLM-5.2 on complex coding and reasoning benchmarks.

Original authors: Zhenyu Hou, Yujiang Li, Jie Tang, Yuxiao Dong

Published 2026-07-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhenyu Hou, Yujiang Li, Jie Tang, Yuxiao Dong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are training a team of student chefs to cook the perfect meal. In the old way of doing things (which the paper calls Synchronous RL), the head chef would shout, "Everyone, start cooking!" and then wait. The students would all start working on their dishes. But here's the problem: some students are fast, and some are slow. The fast students would finish their dishes and stand around staring at the wall, waiting for the slowest student to finish. Only when everyone was done could the head chef taste the food, give feedback, and tell everyone what to do next. This is incredibly inefficient; the kitchen is full of idle time.

To fix this, researchers tried Asynchronous RL. In this version, as soon as a student finishes a dish, the head chef immediately tastes it and gives feedback. The student can start their next dish right away. The kitchen never stops moving. This is much faster.

However, there was a big catch. Because the head chef was tasting dishes from students who started cooking at different times, the "recipe" the students were following was constantly changing. Some students were cooking based on an old recipe, while others were using a new one. This confusion made the training unstable, and the students often got worse at cooking instead of better. Also, the popular method used to grade these students (called GRPO) required the head chef to wait for a group of students to finish before giving a grade, which brought back the waiting problem.

The authors of this paper introduced a new method called SAO (Single-Rollout Asynchronous Optimization). They solved the chaos with three clever tricks:

1. The "Instant Feedback" Rule (Single-Rollout)

Instead of waiting for a group of students to finish a batch, SAO says: "As soon as one student finishes a dish, grade it immediately."

The Metaphor: Imagine a video game where you get a score the second you finish a level, rather than waiting for your whole party to finish. This removes the "waiting for the slowest person" delay.
The Problem it Solves: It stops the "group" from having to wait for the slowest member, keeping the training moving at full speed.

2. The "Strict Safety Net" (Double-Sided Clipping)

Because the students are working so fast and the recipes are changing, there's a risk they might go crazy and try something totally wild and dangerous.

The Metaphor: The authors put up a "fence" around the training. If a student's new idea is too different from what the teacher expects (too far left or too far right), the system doesn't just ignore it; it completely blocks the student from learning from that specific mistake. It's like a strict coach who says, "If you try to run backward, I won't let you learn from that run at all."
The Problem it Solves: This stops the training from becoming unstable or "exploding" when the students get confused by the fast pace.

3. The "Super-Coach" (Better Value Model)

In the old "group" method, the coach could compare a student's dish to their classmates' dishes to see if it was good. But in this "one-by-one" method, there are no classmates to compare to. The coach needs to be incredibly smart to know if a single dish is good or bad on its own.

The Metaphor: The authors trained a special "Value Coach" (a Critic) who is much smarter and updates their knowledge twice as fast as the student chefs. They also froze the coach's "instincts" (attention layers) so they wouldn't get confused, only letting the coach learn the specific details of the recipe.
The Problem it Solves: This ensures that even when the student is working alone, the coach can accurately tell them, "Yes, that was a good move," or "No, that was bad," without needing a group to compare against.

The Results

The paper tested this new method on two very hard tasks:

Coding: Asking the AI to fix bugs in software (like a mechanic fixing a car).
Math Reasoning: Asking the AI to solve complex math problems (like a student taking a hard exam).

The Outcome:

The old method (GRPO) would start well but then crash and fail after a while because it got too confused.
The new SAO method kept training smoothly for a long time (up to 1,000 steps) and consistently got better scores.
It also proved that this method is perfect for Online Learning, where the rules of the game change while you are playing. For example, if the teacher suddenly says, "Now I want cute stories," the SAO-trained AI could quickly switch its style, whereas other methods were too slow to adapt.

In short, the paper says: "Stop waiting for groups. Let everyone work at their own pace, but give them a strict safety net and a super-smart coach who updates their knowledge instantly. This makes AI training faster, more stable, and better at solving hard, changing problems."

Technical Summary: Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning

Problem Statement

Reinforcement Learning (RL) has become a critical component in the post-training of Large Language Models (LLMs), particularly for agentic tasks involving long horizons, coding, and complex reasoning. However, existing RL pipelines are predominantly synchronous and batch-interleaved. In these systems, the policy generates a batch of rollouts, and optimization only begins after the entire batch is collected. This approach is inefficient for agentic workloads where trajectory lengths vary significantly; short trajectories finish quickly while long ones act as stragglers, causing substantial GPU cluster idle time.

While asynchronous RL—which updates the model continuously as rollouts arrive—offers a solution to improve utilization and wall-clock efficiency, it introduces two primary challenges:

Training Instability and Off-Policy Drift: In asynchronous settings, rollouts are generated by multiple older versions of the policy model. This "policy lag" creates unpredictable off-policy behavior that can destabilize training.
Incompatibility with Group-Wise Sampling: Widely used algorithms like Group Relative Policy Optimization (GRPO) rely on sampling a group of responses per prompt to estimate advantages via group-level averaging. This group-wise approach is mismatched with asynchronous training because the group must wait for the slowest member to finish, reintroducing latency and off-policy bias. Furthermore, group-wise sampling is ill-suited for online or complex agentic settings where the environment often provides only a single trajectory feedback per prompt.

Methodology: Single-Rollout Asynchronous Optimization (SAO)

The authors propose SAO, a framework designed to stabilize asynchronous RL training while preserving its efficiency. SAO replaces group-wise sampling with single-rollout sampling (one rollout per prompt) and introduces specific mechanisms to handle the resulting off-policy and variance challenges.

1. Direct Double-Sided Importance Sampling (DIS)

To address the "policy lag" between the rollout engine and the training model without the computational cost of tracking a history of old policies:

Simplified Ratio Calculation: SAO directly uses the rollout policy ( $\pi_{rollout}$ ) as the behavior proxy and the current policy ( $\pi_\theta$ ) for importance sampling, effectively dropping the need for a separate, potentially stale "old policy" ( $\pi_{old}$ ). The probability ratio is computed directly from rollout logs: $r_t(\theta) = \exp(\log \pi_\theta - \log \pi_{rollout})$ .
Double-Sided Token-Level Clipping: Unlike standard PPO clipping, SAO employs a strict double-sided clipping strategy. It restricts the trust region to $[1-\epsilon_\ell, 1+\epsilon_h]$ . Tokens falling outside this range are masked entirely from gradient computation. This aggressive masking prevents instabilities caused by extreme policy divergence.

2. Single-Rollout Sampling with Enhanced Value Modeling

Replacing group-wise sampling with single-rollout updates reduces latency but increases gradient variance. To mitigate this, SAO introduces specific value-model training strategies:

Faster Value Updates: The value network ( $V_\phi$ ) is updated more frequently than the policy ( $\pi_\theta$ ). Specifically, for every single policy gradient update, the value network undergoes $K$ updates (where $K=2$ in experiments). This ensures value estimates adapt quickly to the current policy before being used for advantage calculation.
Frozen-Attention Training: To stabilize the value model, the authors observe that instability primarily originates from Full Attention layers. Consequently, they freeze the attention parameters in the value model and only optimize the Mixture-of-Experts (MoE) projections. This regularizes the value model, leveraging pre-trained attention weights while allowing the model to learn task-specific value functions.
Skip-Observation Token-Level GAE: For agentic tasks with interleaved actions ( $a_i$ ) and environment observations ( $o_i$ ), standard Generalized Advantage Estimation (GAE) introduces noise when calculating advantages across the boundary where the model does not generate the observation. SAO derives a Skip-Observation GAE that links the value of the current action directly to the value of the next action, bypassing the external environment feedback tokens.

3. Value Model Pretraining Scaling

The authors emphasize that a robust initialization is critical. They significantly scale the data used for value model pretraining to address the "cold start" problem, ensuring the value model is effective from the early stages of single-rollout training.

Key Contributions

Stabilization Mechanism: Introduction of a simplified yet aggressive token-level importance sampling strategy (DIS) that eliminates the need for historical policy tracking while enforcing strict double-sided clipping to maintain training stability.
Single-Rollout Paradigm: A shift from group-wise to single-rollout sampling to eliminate synchronization barriers in asynchronous RL, coupled with specific value-model training designs (faster updates, frozen attention) to manage the resulting variance.
Agentic-Specific Estimator: Derivation of a Skip-Observation token-level GAE estimator that correctly handles the discontinuity between model-generated actions and environment feedback in multi-turn agent trajectories.
Empirical Validation: Demonstration that SAO can train stably for over 1,000 steps, outperforming GRPO and its variants on diverse benchmarks.

Experimental Results

SAO was evaluated on agentic coding and math reasoning benchmarks, including SWE-Bench Verified, AIME2025, BeyondAIME, HMMT Nov 2025, and IMOAnswerBench.

Performance: SAO consistently outperformed the baseline (Qwen3-30B-A3B SFT) and GRPO variants across all benchmarks.
- On AIME2025, SAO achieved 97.3% accuracy compared to 84.2% for GRPO (w/ python) and 80.4% for the SFT baseline.
- On SWE-Bench Verified, SAO achieved 29.8% accuracy, surpassing GRPO (27.0%) and the baseline (23.0%).
Stability: While standard GRPO suffered from performance collapse around 160 training steps, SAO maintained stable performance and continued to improve up to 1,000 steps.
Ablation Studies: Removing components such as "Faster Value Updates" or "Frozen Attention" resulted in performance degradation, confirming the necessity of these design choices.
Online Learning: In a simulated online learning setting with shifting reward preferences (writing styles), SAO demonstrated rapid policy realignment, whereas a Running Mean baseline exhibited significant adaptation lag.

Significance and Claims

The paper claims that SAO successfully addresses the trade-off between efficiency and stability in asynchronous RL for LLMs. By moving away from the group-wise sampling inherent in GRPO, SAO enables true asynchronous training where rollouts are consumed immediately upon arrival, eliminating straggler effects.

The authors assert that their approach is particularly effective for agentic tasks and online learning scenarios where single-trajectory feedback is the norm. The successful deployment of SAO in the training pipeline for the open GLM-5.2 model (750B-A40B) serves as a practical validation of the method's scalability and robustness in real-world, large-scale settings. The work suggests that with proper value-model design and off-policy correction, asynchronous RL can be a stable and superior alternative to synchronous batch methods for complex, long-horizon reasoning tasks.