Technical Summary: RC-GRPO for Multi-Turn Tool Calling Agents

1. Problem Statement

Multi-turn tool calling presents significant challenges for Large Language Models (LLMs) due to the sparsity of rewards and the high cost of exploration. While Group Relative Policy Optimization (GRPO) is a popular reinforcement learning (RL) method that eliminates the need for a separate critic network, it suffers from a critical failure mode when applied to multi-turn tool agents initialized with Supervised Fine-Tuning (SFT).

The core issue is the "paradox of perfection." When an LLM is SFT-trained on optimal demonstrations, it develops a highly peaked policy (a strong "golden-path" prior). During GRPO training, this leads to low diversity in rollout groups; multiple sampled trajectories often receive identical rewards (e.g., all 0 or all 1). Consequently, the within-group standard deviation ($\sigma_g$) approaches zero, causing the group-normalized advantage signal to degenerate ($A_j \approx 0$). This results in vanishing gradients and stalled learning, particularly in partially observable environments where local confidence can lead to repetitive, short-horizon behaviors.

2. Methodology: RC-GRPO

The authors propose RC-GRPO (Reward-Conditioned Group Relative Policy Optimization), a two-stage pipeline designed to treat within-group diversity as a controllable variable rather than a byproduct of sampling temperature.

Stage 1: Reward-Conditioned Trajectory Policy (RCTP) Finetuning

Before RL, the base LLM is fine-tuned into a Reward-Conditioned Trajectory Policy (RCTP).

• Data Construction: A mixed-quality dataset is created by pairing trajectories with discrete reward goal tokens (e.g., <|high reward|> for success, <|low reward|> for failure).
• Objective: The model is trained to maximize the log-likelihood of actions conditioned on both the history and the specific reward token: $\pi_\theta(a_t | h_t, r)$.
• Outcome: The policy learns to generate distinct quality trajectories on demand. Conditioning on <|high reward|> elicits optimal actions, while <|low reward|> generates plausible failure modes.

Stage 2: Reward-Conditioned GRPO

The RCTP serves as the initialization for the RL stage.

• Controlled Sampling: For each prompt, a group of $G$ trajectories is generated. Unlike standard GRPO, which samples from a single unconditional policy, RC-GRPO samples a discrete reward token $r_j$ for each trajectory in the group from a fixed distribution $P_{sample}(r)$.
• Conditioned Rollout: Each trajectory $\tau_j$ is generated conditioned on its specific token $r_j$ (i.e., $a_{t,j} \sim \pi_\theta(\cdot | h_{t,j}, r_j)$).
• Advantage Calculation: Because the group contains trajectories conditioned on different reward modes (high vs. low), the within-group reward variance is explicitly injected. This ensures that the group-normalized advantage $A_j = \frac{R(\tau_j) - \mu_g}{\sigma_g + \epsilon}$ remains non-degenerate and informative.
• Reward Function: A unified trajectory-level reward $R(\tau)$ is used, defined as the product of state consistency ($R_{state}$) and essential action coverage ($R_{action}$), resulting in a binary success signal (1 for success, 0 for failure).

3. Key Contributions

1. RC-GRPO Framework: The introduction of a reward-conditioned approach that transforms exploration into a controllable steering problem via discrete reward tokens, preventing the vanishing advantage problem in peaked policies.
2. Two-Stage Pipeline: A specific workflow involving RCTP fine-tuning on mixed-quality data followed by reward-conditioned GRPO, which ensures the policy can reliably produce diverse behaviors required for effective group-relative updates.
3. Empirical Validation: Comprehensive experiments on the Berkeley Function Calling Leaderboard v4 (BFCLv4) multi-turn benchmark using LLaMA-3.1-8B-Instruct and Qwen2.5-7B-Instruct.
4. Theoretical and Dynamic Analysis:
   • Theoretical: A variance-based analysis (Propositions 4.2 and 4.3) demonstrating that standard GRPO collapses when $\sigma_g \to 0$ due to peaked SFT policies, while reward conditioning guarantees a lower bound on within-group variance.
   • Dynamic: Evidence showing that RC-GRPO improves performance while decreasing policy entropy, proving that gains are driven by structured variance injection rather than increased randomness.

4. Experimental Results

The method was evaluated on the BFCLv4 multi-turn split.

• Qwen2.5-7B-Instruct: The full RC-GRPO pipeline achieved 85.00% overall accuracy. This significantly outperformed the Base Model (11.25%), standard SFT+GRPO (48.75%), and even surpassed all evaluated closed-source API models (the best of which, Opus-4.5, achieved 61.25%).
• LLaMA-3.1-8B-Instruct: The method achieved 48.75% accuracy, improving over SFT+GRPO (35.00%) and the Base Model (0.00%).
• Ablation Studies:
  • Necessity of RCTP: Adding reward conditioning to a standard SFT initialization yielded negligible gains. However, switching the initialization to RCTP-FT under RC-GRPO resulted in massive improvements (+38.75% on Qwen2.5-7B).
  • Mechanism: Analysis of training dynamics revealed that RC-GRPO achieves high performance with lower entropy and a negative correlation between entropy and reward, contrasting with standard GRPO where performance relies on high entropy.

5. Significance and Claims

The paper claims that RC-GRPO addresses a fundamental limitation in applying group-based RL to strong, SFT-initialized agents: the collapse of the learning signal due to lack of within-group diversity.

• Controlled Diversity: The work demonstrates that diversity in RL rollouts can be explicitly controlled via conditioning tokens rather than relying on stochastic sampling temperature, which is often insufficient for peaked policies.
• Stability over Randomness: The authors argue that the observed improvements are not due to increased randomness (entropy) but rather to the restoration of informative, non-vanishing advantage signals through structured variance.
• Practical Impact: By enabling open-weight models to outperform closed-source API models on complex multi-turn tool calling tasks, the method offers a scalable path for training robust agentic systems without the computational overhead of critic networks or the instability of standard GRPO.

The authors maintain a modest stance, noting that while they provide a variance-based theoretical explanation for the method's stability, they do not claim a complete theory of multi-turn agent training dynamics. The primary contribution is framing controllable generation and group-based RL as a unified solution to the "vanishing update" problem in sparse reward environments.