Technical Summary: SAPO (Single-Rollout Autoregressive Policy Optimization)

1. Problem Statement

Agentic Reinforcement Learning (RL) has emerged as a critical post-training stage for Large Language Models (LLMs), enabling them to perform long-horizon interactive tasks such as reasoning, tool use, and web navigation. While existing methods have achieved success, they face three primary limitations:

1. Lack of Explicit Value Generalization: Critic-free, group-relative methods (e.g., GRPO) estimate advantages by normalizing rewards across multiple sampled trajectories. This approach fails to learn a generalized value function that can assign credit to specific states or prompts independently of the group, leading to weak temporal credit assignment.
2. Advantage Collapse: In long-horizon tasks with sparse feedback, it is common for all rollouts in a group to receive identical or near-identical rewards. This causes the normalized advantages to vanish, resulting in zero-gradient regions or "advantage collapse," which stalls learning.
3. Inefficient Trade-offs: Group-relative methods require a costly trade-off between sampling budget and performance. Small groups yield noisy baselines, while large groups improve stability at the expense of massive computational costs and synchronization barriers. Conversely, traditional Proximal Policy Optimization (PPO) with a learned critic offers explicit value learning but incurs substantial memory and computational overhead due to the need for a separate, policy-scale critic model.

Current attempts to mitigate these issues (e.g., DAPO, ReMax, VinePPO) either remain dependent on multiple rollouts, retain coarse credit assignment, or require separate value networks that duplicate parameters. There is a need for a framework that supports efficient single-rollout learning with explicit temporal value estimation without maintaining a separate critic.

2. Methodology: SAPO

The authors propose Single-Rollout Autoregressive Policy Optimization (SAPO), a low-memory and compute-efficient framework that unifies policy and value learning within a single autoregressive backbone.

Core Architecture: Autoregressive Actor-Critic

SAPO leverages the causal structure of LLMs to represent the policy, state value ($V$), and action value ($Q$) within a single model stream.

• Causal Boundaries: The method identifies two distinct causal boundaries within an agent's turn:
  1. Pre-action boundary: The state context ($c^s_t$) before the action is generated.
  2. Post-action boundary: The state context concatenated with the generated action ($c^{s,a}_t$).
• Shared Backbone: The same transformer model generates the action tokens between these boundaries.
• Two-Token Value Basis: Instead of a separate value head, SAPO reserves two specific vocabulary tokens ($w^+$ and $w^-$) exclusively as a value basis. The model reads the logits of these two tokens at the respective causal boundaries to compute normalized scalar values:
  • $V_\theta(s_t)$ is read at the pre-action boundary.
  • $Q_\theta(s_t, a_t)$ is read at the post-action boundary.
  • The value is derived from the difference in logits, clipped to a normalized range $[-1, 1]$, and scaled by the maximum possible return ($R_{max}$).
• Policy Separation: The reserved value tokens are excluded from the action vocabulary during policy generation and probability normalization, ensuring they do not interfere with the action distribution.

Single-Rollout Advantage Estimation

Unlike group-relative methods, SAPO samples only one trajectory per task.

• Trajectory-Level GAE: The method employs a Generalized Advantage Estimator (GAE) that propagates delayed terminal rewards backward through the trajectory using the learned state values ($V$).
• Targets:
  • The State Value is trained toward the $\lambda$-return (propagating long-range outcome information).
  • The Action Value is trained toward a on-policy SARSA target (grounding the post-action boundary in immediate consequences).
• Batch Normalization: To stabilize training without group statistics, turn-level advantages are normalized using batch statistics ($\mu_B, \sigma_B$) across the current training batch before being broadcast to all tokens in the action.

Joint Optimization

SAPO optimizes a unified objective function in a single backward pass:

1. Token-Level Policy Loss: A clipped PPO-style loss using the batch-normalized advantages.
2. Value Losses: Clipped regression losses for both $V$ and $Q$ objectives, averaged over turns rather than tokens to ensure equal weight for each decision regardless of response length.
3. Regularization: Includes KL divergence against a reference policy and entropy regularization.

3. Key Contributions

1. Unified Architecture: Introduction of a single-rollout autoregressive actor-critic architecture that jointly represents and trains policy, state value, and action value through causal-boundary readouts, eliminating the need for a separate critic network.
2. Novel Estimator: Development of a trajectory-level advantage estimator combining $\lambda$-returns with batch normalization, enabling explicit turn-level credit assignment without relying on group-relative sampling.
3. Efficiency and Stability: Demonstration that this unified framework achieves a favorable balance of performance, training stability, memory efficiency, and runtime efficiency on long-horizon agentic tasks.

4. Experimental Results

The authors evaluated SAPO on the ALFWorld (embodied reasoning) and WebShop (web navigation) benchmarks using Qwen2.5-1.5B and Qwen2.5-7B backbone models.

• Performance: SAPO consistently outperformed standard PPO and GRPO.
  • On ALFWorld with Qwen2.5-1.5B, SAPO achieved 90.1% aggregate success, improving over PPO and GRPO by +35.7 and +17.3 percentage points, respectively.
  • On WebShop, SAPO achieved a score improvement of +8.4 points and a success rate increase of +12.2 points over PPO.
  • Across the experiments (as summarized in the Abstract), SAPO trains stably and outperforms PPO and GRPO by mean +15.1 and +12.1 percentage points, respectively.
  • With the 7B model, SAPO obtained 94.0% ALFWorld success and an 88.6 WebShop score with 82.4% success, outperforming both PPO and GRPO as well as recent variants on all three aggregate metrics.
• Efficiency:
  • Memory: SAPO eliminates the memory footprint of a separate critic model (which is typically policy-scale).
  • Runtime: SAPO reduced the per-iteration runtime by 33.2% compared to PPO. The primary savings came from removing the separate value inference and critic optimization stages, reducing total time from 451.2s to 301.4s per iteration on ALFWorld.
• Stability: The method trained stably from a single rollout per prompt, avoiding the advantage collapse issues seen in group-relative methods under sparse feedback.

5. Significance and Claims

The paper claims that SAPO resolves the tension between the explicit value generalization of actor-critic algorithms and the memory/computational efficiency of critic-free, group-relative optimization.

• Architectural Alignment: The authors argue that the information flow required for actor-critic learning naturally aligns with the causal order of language generation. By exploiting this alignment, SAPO enables efficient value-based optimization within a single causal model.
• Scalability: The framework demonstrates that explicit value learning does not require duplicated models or costly sampling, offering a viable path toward efficient long-horizon agentic RL.
• Generalization: The results suggest that the architectural alignment between autoregressive modeling and actor-critic learning can reduce dependence on task-specific dense reward engineering, as the method performs well with sparse outcome rewards.

The paper concludes that SAPO represents a significant step forward in making agentic RL more scalable and practical for complex, multi-turn interactions.