Technical Summary: PlanPO – Group Planning-Aware Policy Optimization for Multi-Turn Agentic LLMs

Problem Statement

In the domain of multi-turn agentic Large Language Models (LLMs), Group Relative Policy Optimization (GRPO) has emerged as a standard paradigm for training agents using verifiable outcome rewards. However, a critical limitation exists in how GRPO handles successful rollout heterogeneity.

Existing variants typically assign identical outcome rewards to all trajectories that successfully complete a task, regardless of the efficiency of the interaction. This leads to "advantage collapse," where:

1. Circuitous successes (trajectories with redundant state transitions, loops, or dead ends) are treated equally with optimal successes.
2. Verbose or logically flawed reasoning within successful responses receives the same credit as concise, coherent reasoning.
3. The lack of discrimination among successful trajectories prevents agents from learning generalizable planning abilities, often causing them to merely imitate specific high-quality patterns rather than developing robust, task-general strategies.

Current solutions attempting to address this often rely on value models (critics) or process reward models (PRMs), which introduce estimation bias, proxy errors, and significant memory overhead. Other methods depend on labor-intensive manual design or empirical heuristics that lack generality across different task settings.

Methodology: PlanPO

The authors propose Group Planning-aware Policy Optimization (PlanPO), a simple yet effective reinforcement learning method that leverages naturally available length profiles to construct coarse-to-fine advantage signals. The core innovation is conditioning length normalization strictly on successful trajectories within a group, thereby avoiding the pitfalls of standard length minimization.

1. Success-Conditioned Length Normalization

PlanPO operates within the standard group-relative framework but introduces two distinct levels of advantage computation, both conditioned on the trajectory achieving the task goal ($R(\tau) = 1$):

• Trajectory-Level Advantage ($A^E$):
  For a group of $N$ sampled trajectories for the same task, PlanPO normalizes the outcome reward by the trajectory length (number of interaction turns, $T_i$) only for successful trajectories.
  $$R^E(\tau^i) = \mathbb{1}[\tau^i \in G^U_x] \cdot \frac{R(\tau^i)}{T_i}$$
  The advantage $A^E$ is computed via group-relative normalization (mean and variance) of these length-normalized rewards. This signal encourages the agent to find shorter, more direct interaction paths among successful rollouts.

• Response-Level Advantage ($A^S$):
  At the turn level, PlanPO flattens all successful responses generated for the same task into a single group. It normalizes the outcome reward by the response token length ($L_{i,t}$) for each turn.
  $$R^S(a_{i,t}) = \mathbb{1}[\tau^i \in G^U_x] \cdot \frac{R(\tau^i)}{L_{i,t}}$$
  The advantage $A^S$ is computed within this active response group. This provides fine-grained supervision, encouraging concise and coherent reasoning traces within successful interactions.

2. Coarse-to-Fine Policy Optimization

The final advantage for a token $a_{i,t}$ in trajectory $\tau^i$ is a weighted combination of the trajectory-level and response-level advantages:
$$A^{\text{PlanPO}}(a_{i,t}) = A^E(\tau^i) + \alpha(k) A^S(a_{i,t})$$

• Weighted Decay: The coefficient $\alpha(k)$ is linearly decayed during training. This design prioritizes the trajectory-level planning signal (coarse) initially and gradually refines it with the response-level signal (fine).
• Bias-Variance Trade-off: Theoretical analysis (Theorem 1) demonstrates that $\alpha(k)$ balances the bias-variance trade-off. A larger $\alpha$ preserves turn-level refinement but increases sampling variance, while a smaller $\alpha$ reduces variance at the cost of shrinkage bias.

The policy is optimized using a clipped group-relative objective similar to GRPO, incorporating a KL-divergence penalty against a reference policy.

Key Contributions

1. Identification of a Bottleneck: The paper identifies that successful-rollout heterogeneity (redundant turns and verbose reasoning in successful trajectories) is a critical bottleneck in group-relative optimization that masks rollout quality and limits policy learning.
2. Proposal of PlanPO: A novel method that constructs success-conditioned length-normalized advantages. Unlike generic length-based reward shaping, PlanPO only applies length normalization within groups of successful rollouts, ensuring agents learn planning-aware behaviors rather than simply minimizing length.
3. Empirical Validation: Extensive experiments demonstrate that PlanPO consistently outperforms strong baselines (including GRPO, GiGPO, and PPO) across multiple benchmarks with negligible additional training cost.

Experimental Results

The authors evaluated PlanPO using Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct on three challenging multi-turn benchmarks: ALFWorld, WebShop, and SciWorld.

• Performance Gains:

  • On average, PlanPO improves over GRPO by 27.2% across the benchmarks.
  • In ALFWorld (1.5B model), PlanPO achieved a 91.3% overall success rate, an improvement of +18.5 points over GRPO.
  • On out-of-distribution (OOD) tasks in ALFWorld, PlanPO achieved an 87.1% success rate, outperforming GRPO by +17.0 points and GiGPO by +4.7 points, indicating strong generalization.
  • In SciWorld, PlanPO-7B achieved an overall score of 68.46, significantly outperforming the previous best RL baseline (ScalingInter-7B at 57.00).

• Efficiency and Behavior:

  • Interaction Efficiency: PlanPO agents reduced the mean number of interaction turns from 26.1 (GRPO) to 13.8.
  • Response Conciseness: Mean response token length decreased from 95.1 (GRPO) to 56.3.
  • Cost: The method incurs negligible additional training cost, as the length normalization and advantage computation are lightweight operations compared to the forward/backward passes of the LLM.

• Ablation Studies:

  • Removing either the trajectory-level ($A^E$) or response-level ($A^S$) advantage significantly degraded performance.
  • Unconditional length shaping (without success conditioning) caused performance to collapse, confirming that length minimization alone is detrimental.
  • Success-conditioned length shaping alone was insufficient; the combination with group-relative optimization was necessary to yield gains.

Significance and Claims

The paper claims that PlanPO offers a simple, scalable, and task-general solution to the problem of advantage collapse in agentic RL. By leveraging the inherent heterogeneity in successful rollouts (specifically interaction and response lengths) as a supervisory signal, PlanPO enables agents to learn generalizable planning abilities rather than overfitting to specific high-quality behavioral patterns.

The authors emphasize that their approach does not require expensive value models, manual reward engineering, or complex heuristics. Instead, it utilizes the "naturally available yet underutilized" signal of length profiles within successful groups to guide agents toward more direct, efficient, and coherent interactions. The results suggest that successful-rollout heterogeneity is a viable and powerful signal for training planning-aware agentic LLMs.