🤖 AI

SynthTools: A Framework for Scaling Synthetic Tools for Agent Development

The paper introduces SynthTools, a fully LLM-based framework that automatically generates, validates, and scales diverse synthetic tool environments and tasks, demonstrating that training agents on this synthetic data significantly improves their tool-use performance on real-world benchmarks.

Original authors: Tommaso Castellani, Naimeng Ye, Daksh Mittal, Thomson Yen, Emmanouil Koukoumidis, William Zeng, Hongseok Namkoong

Published 2026-05-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tommaso Castellani, Naimeng Ye, Daksh Mittal, Thomson Yen, Emmanouil Koukoumidis, William Zeng, Hongseok Namkoong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot how to use a giant, complex toolbox to fix a house. The problem is, you can't just hand the robot a real toolbox with real tools (like a real hammer or a real drill) because:

Real tools are messy: They break, they need batteries, and sometimes they just don't work when you try to use them.
There aren't enough of them: You can't build a training course for every possible tool in the universe using only real ones.
It's hard to make up "practice problems": If you tell the robot, "Fix the leak," it might not know which tools to use or in what order.

SynthTools is a new framework that solves this by building a giant, perfect, virtual simulation factory entirely inside a computer. Instead of using real tools, it uses a super-smart AI (a Large Language Model) to invent, test, and organize thousands of fake tools and practice problems.

Here is how the SynthTools factory works, broken down into three simple steps:

1. The Architect (Top-Down Generation)

Think of this as the Master Designer.
Instead of just randomly scribbling down a list of tools, the AI starts with a big idea, like "Financial Trading" or "E-commerce." It then breaks that big idea down into smaller neighborhoods (sub-domains), then into specific jobs (tasks), and finally invents the specific tools needed for those jobs.

The Analogy: Imagine designing a video game. You don't just throw random items on the ground. You decide, "This is a hospital level," so you invent a "Stethoscope," a "Syringe," and a "Heart Monitor" that all fit together logically. SynthTools does this for 100 different "worlds" (like finance, logistics, healthcare), creating over 73,000 unique tools.

2. The Quality Control Inspector (Simulation & Validation)

This is the Strict Foreman.
Just because the AI invented a tool doesn't mean it works. The Foreman puts every single tool through a stress test.

The Test: The Foreman tries to use the tool in different ways. "What happens if I give you the wrong password?" "What if the flight is fully booked?"
The Result: If the tool gives a confusing answer or crashes, the Foreman throws it in the trash. If it handles the mistake perfectly (e.g., saying "Sorry, no seats left" instead of crashing), it gets a stamp of approval.
The Scale: They tested nearly 80,000 tools. The system is so good at spotting bad tools that it filters out the weak ones, leaving only the reliable 73,883 tools. It's like a factory that only ships products that have passed a 97% accuracy test.

3. The Puzzle Maker (Bottom-Up Task Generation)

Now that they have a pile of approved tools, they need to create practice puzzles for the robot to solve.

The Method: Instead of asking the AI to "Make up a hard problem," they build the problem from the ground up. They take a few approved tools (like a "Return Request" tool and an "Email Sender" tool) and chain them together.
The Result: They create a step-by-step story: "A customer wants to return a shirt. First, check if they are eligible. Second, print a label. Third, email them."
The Magic: Because the AI built the puzzle using the tools it knows work, it also knows the exact solution. It's like a teacher who writes a math test and also writes the answer key at the same time. This ensures every task is solvable and verifiable. They created nearly 80,000 of these puzzles.

The Big Experiment: Does it work?

The researchers took a standard AI model (Qwen3) and gave it a "mid-training" session. Think of this as sending the AI to a driving school using these synthetic puzzles.

The Training: The AI practiced solving these 80,000 fake but realistic problems.
The Test: They then tested the AI on real benchmarks (using real-world APIs and actual function-calling tests).
The Outcome: The AI that practiced on the synthetic factory got significantly better at using real tools. It's as if practicing on a perfect driving simulator made the driver better at handling a real car on a rainy day.

Summary

SynthTools is a pipeline that uses AI to build a massive, self-checking library of fake tools and fake problems. It solves the problem of "we don't have enough real data to train robots" by generating its own high-quality, diverse, and verified training data. The result is a framework that can scale up to millions of tools and tasks, helping AI agents learn how to use tools more effectively, even in the real world.

Technical Summary: SynthTools

Problem Statement

Agentic systems require the ability to use external tools to solve complex, long-horizon tasks. However, current models struggle with reliable tool use, a limitation often attributed to the scale and diversity of training data. Existing approaches to creating tool-use environments face significant bottlenecks:

Real APIs: While authentic, they are difficult to scale due to API keys, rate limits, usage quotas, and frequent interface changes.
Curated Suites: Hand-crafted collections (e.g., ACEBench, $\tau^2$ -bench) are limited in the number of tools and domain diversity, often covering only a handful of broad domains.
Synthetic Generation: While scalable, many existing synthetic pipelines rely on programmatic simulation, which restricts tool complexity to simple CRUD operations. Conversely, LLM-based simulation has been underexplored and often lacks deterministic execution or relies on evolving real API documentation.

Furthermore, even with large tool sets, generating realistic, non-trivial tasks that require coordinated multi-step reasoning remains a fundamental bottleneck. Naive prompting often yields shallow scenarios that fail to demand long-horizon planning or decision-making under uncertainty.

Methodology

The authors introduce SynthTools, a fully LLM-based pipeline designed to span the entire lifecycle of tool-use environment creation: generation, simulation, validation, and task construction. The framework is built on three core modules:

1. Top-Down Environment Generation

Instead of generating flat lists of APIs, SynthTools employs a hierarchical domain evolution procedure to construct domain-grounded tools:

Field $\to$ Sub-domain: An LLM partitions seed fields (e.g., Finance, Healthcare) into coherent subdomains, surfacing stakeholders and entities.
Sub-domain $\to$ Task: The model proposes families of tasks with natural language descriptions suitable for documentation.
Task $\to$ Tool: Concrete tools are instantiated for each task. The generation process encourages composability by specifying upstream dependencies and downstream affordances.
Output: Each tool is defined by a name, natural language description, parameter schema, and an I/O contract (preconditions, postconditions, error modes). This approach allows for tailored complexity, including complex object types and lists, exceeding the scope of simple CRUD operations found in programmatic environments.

2. Environment Simulation and Validation

To ensure reliability, the framework simulates tool behavior and validates the results using an LLM-as-a-Judge:

Simulation: The simulator decomposes execution into parameter validation (using AST parsing for syntax and LLMs for semantic constraints) and response generation. It incorporates metadata to condition responses, ensuring consistency (e.g., returning "no seat available" if a flight is full).
Validation: An LLM judge evaluates tool specifications and simulated responses. It filters out unstable tools and supports retries. The system targets a low false-positive rate, achieving 97% accuracy in identifying incorrect behaviors.
Reliability: The pipeline achieves 94% accuracy in simulating tool responses and filters out tools that cannot be reliably emulated.

3. Bottom-Up Task and Trajectory Generation

Complementing the top-down tool generation, this module constructs tasks from the validated toolset:

Tool-Chain Proposal: For each environment, an LLM proposes sequential tool-use chains that serve as the backbone for candidate tasks.
Mini-Task Generation: The system iteratively constructs mini-tasks. For each step, it generates a task description and a ground-truth tool call, verifies the agent's execution against the ground truth, and updates the environment state.
Trajectory Construction: By chaining these interdependent mini-tasks, the system produces complex, verifiable tasks with explicit solution trajectories.
Tunability: Difficulty is controlled by varying the number of mini-tasks, adjusting the trade-off between explicitness and ambiguity, and adding irrelevant but plausible tools.

Key Contributions

Scalable Framework: SynthTools is a fully automated pipeline that generates diverse, controllable, and complex tool-use environments without reliance on real APIs or manual curation.
Dataset Release: The authors release a dataset comprising:
- 73,883 validated tools across 6,800 environments.
- 100 distinct fields (e.g., Financial Trading, Transportation Logistics, E-commerce).
- 79,925 verifiable tasks with ground-truth trajectories.
- This represents a >2 $\times$ increase in scale over prior collections in terms of domains and tools per domain.
Quality Control: The framework includes rigorous simulation and validation steps, ensuring high reliability (94% simulation accuracy, 97% validation accuracy) and low redundancy (only 9% of tools exceed a similarity threshold of $\tau=0.85$ ).
Midtraining Case Study: The authors demonstrate the utility of the dataset by midtraining Qwen3 models (0.6B, 1.7B, 4B) on the generated trajectories.

Results

The paper evaluates the midtrained Qwen3 models on multiple agentic tool-use benchmarks:

Benchmarks: ACEBench, Berkeley Function Calling Leaderboard (BFCL), and API-Bank.
Performance Gains: Across all model sizes and benchmarks, the midtrained models showed consistent performance improvements over their base counterparts.
- For example, on the 4B model, performance on ACEBench (Normal) improved from 0.583 to 0.651, and on BFCL (Live) from 0.432 to 0.538.
Transferability: Improvements on benchmarks involving real APIs (API-Bank, BFCL Live) suggest that tool-use capabilities trained on synthetic data can transfer to real-world API-style interactions.

Significance and Claims

The authors position SynthTools as a necessary infrastructure to address the scalability and diversity gaps in current agentic training landscapes. They claim that:

Synthetic Data is Viable: Training on high-quality, synthetic tool-use trajectories can effectively bridge the distribution gap between generic pretraining and downstream tool-use tasks.
Scalability is Achievable: The framework enables the generation of tool ecosystems at scales and domain coverages unattainable by real-API or curated approaches.
Structured Generation is Critical: The paper argues that effective task generation requires structured, bottom-up construction rather than naive prompting, which often yields shallow or ungrounded scenarios.

The work suggests that synthetic interaction corpora are a key ingredient for advancing tool-using AI systems, providing a practical resource for large-scale training and evaluation.