Technical Summary: RODS – Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents

1. Problem Statement

The training of Large Language Model (LLM) agents for multi-turn tool use via Reinforcement Learning (RL) faces a critical bottleneck: the rapid depletion of informative samples in static datasets. As agents improve, static datasets suffer from signal depletion (Challenge C2), where the gradient signal vanishes because tasks are either mastered (high reward) or remain unsolvable (low reward), leaving the agent with no "boundary" tasks to drive learning.

Existing approaches fail to address this holistically:

• Large-scale offline synthesis (e.g., APIGen-MT, TOUCAN) generates massive corpora upfront but remains decoupled from the training loop, failing to track the agent's evolving capability boundary.
• Online environment augmentation (e.g., EnvTuning) enriches feedback on fixed data but is ultimately constrained by the diversity of the seed corpus.
• Unconstrained self-play often produces semantically disjointed trajectories (Challenge C3) lacking the long-horizon coherence required for multi-turn reasoning.

Consequently, there is a need for a method that dynamically synthesizes data strictly tracking the agent's capability boundary while maintaining multi-turn semantic coherence, all under conditions of extreme data scarcity.

2. Methodology: RODS Framework

The authors propose RODS (Reward-driven Online Data Synthesis), a closed-loop framework that couples data generation directly with the RL training process. The core insight is that the Progress Reward variance in policy gradient methods (like GRPO) serves as a zero-cost, practical detector for the agent's capability boundary.

2.1 Theoretical Basis: Variance as a Boundary Detector

In GRPO, the gradient signal density is governed by rollout reward variance. Citing Popoviciu's inequality, the variance of a bounded variable is maximized when the mean probability of success ($\mu$) is approximately 0.5. RODS operationalizes this by identifying tasks where the agent's average progress reward ($\bar{r}_i$) falls within a specific boundary zone ($\alpha_- \le \bar{r}_i \le \alpha_+$, typically $[0.20, 0.85]$). These "boundary seeds" represent the frontier where successes and failures are balanced, offering the highest concentration of informative gradient signals.

2.2 Core Components

RODS consists of three co-evolving modules:

1. Reward-Based Seed Detection:

   • The system partitions the task space into Mastered, Boundary, and Hard regions based on average progress rewards across $K$ rollouts.
   • It selects "boundary seeds" ($\bar{r}_i \approx 0.5$) using a type-quota mechanism to ensure diverse skill coverage (e.g., missing functions, long context).
   • Selection prioritizes seeds closest to the capability midpoint ($\mu \approx 0.5$) to maximize gradient potential.

2. Skill-Aligned Data Synthesis:

   • Instead of simple paraphrasing, RODS employs a five-stage multi-agent pipeline to generate structurally isomorphic variants.
   • Structural Preservation: The pipeline extracts a complexity profile $\Phi(x_{seed})$ (e.g., API dependency graphs, parameter flows) from the seed. New tasks are sampled to match this profile in dependency depth and topology, ensuring the new task remains near the capability boundary.
   • Semantic Coherence: A Coherence Rewrite Agent anchors all turns to a single narrative goal, resolving the "semantic disjointedness" (C3) common in naive synthesis.
   • Feedback-Driven Execution: An Execution Orchestrator runs the plan in a simulated environment. If errors occur, a Config Patch Agent modifies the environment state (e.g., fixing authentication or resource availability) to ensure valid execution, rather than discarding the sample.

3. Dynamic Replay Buffer Management:

   • Staged Injection: Synthesized variants are held in a candidate queue and injected at epoch boundaries to prevent distributional shock.
   • Multi-Layer Retirement: To maintain pool informativeness, the system employs a three-layer retirement mechanism:
     • Burn-in filtering: Discards variants with initial rewards below a threshold.
     • Boundary-drift eviction: Removes tasks that have drifted into mastered or unsolvable zones.
     • Variance-prioritized pruning: If the pool exceeds capacity ($P_{max}$), samples with lower reward variance are retired to preserve high-gradient signals.

3. Key Contributions

1. RODS Framework: A reward-driven boundary expansion method that repurposes the RL progress reward to identify and expand boundary tasks in real-time, addressing signal starvation in static datasets.
2. Skill-Aligned Synthesis Pipeline: A novel approach that preserves the functional dependency structure of verified seeds while generating novel narratives. This ensures multi-turn semantic coherence and prevents overfitting to static execution paths.
3. Data Efficiency: Demonstrating that starting from only 400 human seeds and maintaining an active pool of $\sim$800 samples, RODS achieves performance comparable to a 17K-sample offline pipeline (using roughly 20$\times$ fewer trajectories).

4. Experimental Results

The authors evaluated RODS on the BFCL V3 multi-turn benchmark (800 samples, 400 for training) using Qwen3-4B-Instruct, Qwen2.5-7B-Instruct, and Llama-3.1-8B-Instruct.

• Performance vs. Baselines: RODS outperformed both "Static Dataset" training and "EnvTuning" (environment augmentation) across all model families. On Qwen3-4B, RODS achieved 56.00% overall accuracy, surpassing the static baseline (50.00%) and EnvTuning (50.50%).
• Data Efficiency: RODS matched the performance of FunReason-MT-4B (trained on 17K offline trajectories) while using an active pool of only $\sim$800 samples.
• OOD Generalization: RODS showed consistent improvements on out-of-distribution benchmarks (BFCL V4, $\tau^2$-bench, ACEBench), suggesting the model learned generalizable reasoning patterns rather than in-distribution memorization.
• Ablation Studies:
  • Removing the coherence rewrite stage caused the largest performance drop ($\downarrow 5.13\%$), highlighting the importance of semantic grounding.
  • Replacing boundary-targeted seed selection with random selection resulted in a significant drop ($\downarrow 4.75\%$), confirming that targeting the boundary is more critical than simply adding data volume.
  • Disabling the retirement mechanism ($\downarrow 3.38\%$) confirmed that accumulating mastered data dilutes the gradient signal.

5. Significance and Claims

The paper claims that RODS effectively recasts the data scarcity bottleneck as a dynamic curriculum design problem. By leveraging the inherent variance of the progress reward as a boundary detector, RODS sustains informative policy gradients throughout training without requiring massive offline corpora.

The authors emphasize that where data is synthesized (at the capability boundary) matters more than how much data is synthesized. The framework achieves a favorable trade-off: while it incurs additional synthesis compute costs (roughly 1$\times$ the training cost in GPU-hours), it reduces the required data volume by an order of magnitude compared to massive offline pipelines.

Limitations: The authors note that the current synthesis pipeline relies on deterministic simulation environments (executable Python objects) to verify correctness. Adapting this to opaque environments or remote Model Context Protocol (MCP) servers remains a future direction. Additionally, the system currently uses a single synthesis backbone, though multi-backbone ensembles are suggested as a potential scaling direction.