From Self-Evolving Synthetic Data to Verifiable-Reward RL: Post-Training Multi-turn Interactive Tool-Using Agents

This paper introduces EigenData, a unified framework that combines a self-evolving multi-agent system for synthesizing verifiable tool-use dialogues with a verifier-based reinforcement learning recipe, enabling scalable post-training of interactive agents that achieve state-of-the-art performance on complex multi-turn benchmarks without relying on expensive human annotation.

The Gist

Imagine you are trying to teach a very smart robot butler how to handle complex, real-world chores like booking flights, managing bank accounts, or fixing phone plans. The catch? The robot has to talk to a human customer, figure out what they actually want (which might be vague or changing), and then use a bunch of digital tools to get the job done.

This paper, titled "From Self-Evolving Synthetic Data to Verifiable-Reward RL," presents a new way to train these robots so they don't just guess, but actually learn to succeed.

Here is the story of how they did it, broken down into simple parts with some creative analogies.

The Problem: The "Chaotic Dinner Party"

Training these robots is hard because real life is messy.

1. The Data Problem: To teach a robot, you need thousands of examples of good conversations. But hiring humans to write every single example is slow and expensive.
2. The Simulation Problem: When you train a robot, you can't always use a real human. You use a "simulated human" (a computer program pretending to be a person). The problem is, these simulated humans are often terrible actors. They might forget what they said, make up fake rules, or act irrationally. If the robot learns from a bad actor, the robot learns bad habits.

The Solution: A Two-Part Training Camp

The authors built a system called AReaL-SEA (a fancy name for their training engine) that works in two main phases.

Phase 1: The "Self-Improving Scriptwriter" (AReaL-SEA)

Instead of hiring humans to write the training scripts, they built a team of AI agents that write and critique their own scripts.

• The Analogy: Imagine a theater troupe where the actors, the director, and the critics are all AI.
  • The Scriptwriters create a new play (a task, like "Book a flight for a confused tourist").
  • The Critics read the script and check if it makes sense. If the script is boring or impossible, they send it back.
  • The Actors perform the play. If they mess up, the system notes why.
  • The Evolution Loop: The system looks at all the failures. It says, "Oh, the scriptwriters keep forgetting to include the tourist's passport number," or "The critics are too harsh." It then updates its own instructions to write better scripts and be fairer critics next time.

This creates a "self-evolving" cycle. The system gets better at making high-quality training data without needing a human to hold its hand. Crucially, every time it generates a task, it also builds a checklist (a verifier) to prove if the robot actually solved the problem correctly.

Phase 2: The "Strict Coach" (Verifiable-Reward RL)

Once they have these perfect scripts and checklists, they train the robot using Reinforcement Learning (RL).

• The Analogy: Think of this as a sports coach training an athlete.
  • The User Simulator: Before the robot plays the game, the coach first trains the "fake fans" (the simulated users) to act like real people. If the fans are crazy, the robot gets confused. So, they fine-tune the fans to be realistic.
  • The Game: The robot plays thousands of games against these fans.
  • The Reward: In the past, robots got a "thumbs up" or "thumbs down" based on a vague feeling. Here, the robot gets a binary score based on the checklist. Did the flight get booked? Yes/No. Did the money get deducted correctly? Yes/No.
  • The Group Dynamic: They use a method called GRPO. Imagine a classroom where the teacher doesn't just grade you against a curve, but compares your answer to your classmates. If your answer is better than the average of the group, you get a boost. This helps the robot learn faster even when the "fans" (users) are unpredictable.

The Results: From Novice to Pro

They tested this on three difficult domains: Airline (booking/canceling flights), Retail (shopping), and Telecom (fixing phone bills).

• Before: The robots were okay, but often failed when users got tricky or changed their minds.
• After: Using their self-evolving data and strict coaching, the robots became superstars.
  • On the Telecom test, their robot went from a 28% success rate to a 98.3% success rate.
  • On the Airline test, they beat or matched the most expensive, proprietary models from companies like Google and OpenAI.

Why This Matters

This paper is a game-changer because it shows we don't need to pay millions of dollars to hire humans to write training data. Instead, we can build a self-improving machine that writes its own homework, grades its own tests, and learns from its mistakes.

It's like teaching a child not by giving them a textbook, but by putting them in a room where they can practice, fail, get a clear "correct/incorrect" signal, and automatically get better instructions for the next round. The result is a robot that can handle the chaos of real human conversation with tools, all without breaking the bank.

Technical Summary

Here is a detailed technical summary of the paper "From Self-Evolving Synthetic Data to Verifiable-Reward RL: Post-Training Multi-turn Interactive Tool-Using Agents."

1. Problem Statement

The paper addresses the challenge of post-training Large Language Models (LLMs) to become effective interactive tool-using agents. Unlike static question-answering or single-turn tool use, these agents must operate in a multi-turn dialogue with an active user and an external environment (APIs, databases).

The authors identify two primary bottlenecks in scaling these agents:

1. Scalable Data Acquisition: High-quality multi-turn training data involving complex domain constraints, user preferences, and private information is difficult to obtain. Human annotation is too expensive, and automated synthesis often fails to generate sufficiently challenging tasks or coherent simulated user behaviors.
2. Reinforcement Learning (RL) Instability: Training interactive agents via RL requires a user simulator. Off-the-shelf models often exhibit unstable behavior when simulating users who invoke tools or provide incremental/contradictory information. This introduces noise into the rollout process, leading to corrupted reward signals (e.g., the agent is penalized for a task failure caused by the simulator's error, not the agent's).

2. Methodology

The authors propose a unified framework called AReaL-SEA (Self-Evolving Data Synthesis) combined with a specialized RL recipe for interactive agents.

A. AReaL-SEA: Self-Evolving Data Synthesis

This is a hierarchical multi-agent engine designed to autonomously generate, verify, and refine training data with minimal human supervision.

• Meta-Planning: A meta-planning LLM generates diverse synthesis and evaluation plans covering different domains (e.g., Airline, Telecom), complexity levels, and interaction styles.
• Agent Pipeline:
  1. Task Synthesis: Generates structured candidate tasks (user instruction + task spec + expected answer).
  2. Task Verification: Validates tasks against domain rules and evaluation rubrics.
  3. Trajectory Rollout: Simulates multi-turn interactions between an assistant agent and a user simulator.
  4. Trajectory Verification: Evaluates the full trajectory for success and provides root-cause attribution (distinguishing between task specification errors and agent execution errors).
• Reflection & Self-Evolution: A reflection module analyzes failure patterns from the verification stages. It iteratively updates the synthesis plans and evaluation rubrics to reduce future failures, creating a closed-loop system that improves data quality over time.
• Output: The system produces not only dialogue trajectories but also executable per-instance checkers (verifiers) that serve as ground-truth reward signals.

B. Reinforcement Learning Recipe

The paper adapts Group Relative Policy Optimization (GRPO) for interactive tool use, introducing specific stabilizations:

1. User Model Fine-Tuning: Before RL, the user simulator model is Supervised Fine-Tuned (SFT) on the synthetic data generated by AReaL-SEA. This ensures the user simulator follows instructions reliably and does not introduce noise (e.g., calling wrong tools) that would corrupt the agent's learning signal.
2. Verifier-Based Outcome Rewards: Instead of relying on LLM judges, the system uses the executable checkers generated during data synthesis. The final state of the trajectory is compared against the ground-truth state to determine a binary success/fail reward.
3. Dynamic Filtering: During GRPO training, tasks where all sampled trajectories in a group have identical outcomes (all succeed or all fail) are filtered out. This prevents the calculation of zero advantage, ensuring the model only learns from tasks with meaningful variation in outcomes.
4. Large Batch Sizes: The authors utilize large batch sizes to stabilize advantage estimation in the face of user-driven variance.

3. Key Contributions

1. AReaL-SEA System: A novel self-evolving multi-agent framework that generates verifiable, complex, and high-quality multi-turn tool-use training instances without heavy human annotation.
2. Stabilized RL Recipe: A specific training pipeline for interactive agents that includes user model SFT, dynamic filtering, and verifier-based rewards to mitigate the noise inherent in user simulation.
3. State-of-the-Art Performance: The framework achieves competitive or superior results compared to frontier proprietary models (e.g., GPT-5, Claude Sonnet, Gemini) on the $\tau^2$-bench, using fully open-weight models (Qwen3).

4. Experimental Results

The framework was evaluated on the $\tau^2$-bench across three domains: Airline, Retail, and Telecom.

• Models Tested: Qwen3-30B-A3B and Qwen3-235B-A22B.
• Performance Gains:
  • SFT Impact: Applying SFT with AReaL-SEA data significantly boosted baseline performance. For example, on the Telecom domain, the Qwen3-30B model improved from 28.5% to 85.4% ($pass^1$).
  • RL Impact: Further RL training yielded consistent improvements. The Qwen3-235B model achieved 73.0% on Airline and 98.3% on Telecom, matching or exceeding frontier models like Gemini 3.0 Pro and GPT-5.
  • Mix Training: Training on a mix of all three domains allowed a single model to achieve an average 81.3% $pass^1$, surpassing the performance of separate training on smaller models and rivaling large proprietary models.
• Ablation Studies:
  • User Model Quality: Training with a base (unfined) user simulator degraded performance by ~20% compared to using an SFT-tuned simulator, confirming the critical need for a stable user model.
  • Data Quality: Removing the self-evolution loop or validation agents from AReaL-SEA resulted in significant performance drops, highlighting the necessity of iterative refinement and verification.
  • Algorithm: Dynamic filtering and large batch sizes were shown to be essential for stable GRPO convergence.

5. Significance

This work provides a scalable pathway for bootstrapping complex, long-horizon tool-using behaviors without relying on expensive human annotation. By combining self-evolving synthetic data generation with stabilized, verifier-based RL, the authors demonstrate that open-weight models can be post-trained to match or exceed the capabilities of proprietary frontier models in interactive, multi-turn environments. The approach specifically addresses the "user simulation noise" problem, a critical hurdle in training agentic systems, offering a robust solution for domains like customer support and workflow automation.