Technical Summary: ToolWeave

Problem Statement

The evolution of Large Language Models (LLMs) into autonomous agents relies heavily on multi-turn tool calling, the ability to chain tools, elicit user information, and maintain state across complex workflows. While fine-tuning on synthetic data is a standard approach to enhance these capabilities, existing synthetic data generation pipelines (e.g., ToolFlow, Magnet, ToolDial) suffer from two fundamental limitations that produce unrealistic training data:

1. Semantically Incoherent Workflows: Existing methods often construct tool graphs by inferring edges based on superficial parameter name or description matching (e.g., connecting create_support_ticket to cancel_movie_ticket simply because both have a ticket_id parameter). This results in tool sequences that are syntactically compatible but semantically disjoint, rarely requiring the output of one tool to meaningfully drive another. Consequently, synthesized dialogues lack genuine multi-step interactions; analysis shows fewer than 7% of dialogues from prior pipelines contain true multi-step turns.
2. Parameter Hallucinations: Current approaches typically prompt an LLM to generate an entire dialogue or plan in a single pass, leaving the execution structure implicit. Models must infer argument provenance (whether a parameter comes from the user, context, or a prior tool call) during generation. This leads to high rates of hallucinated arguments, where parameters are neither provided by the user nor produced by preceding tool calls. Prior approaches exhibit hallucination rates exceeding 50% in parameters and tool names.

Methodology: ToolWeave Framework

To address these issues, the authors introduce ToolWeave, a modular, structured framework for synthesizing realistic multi-turn, multi-step tool-calling dialogues. Unlike end-to-end generation, ToolWeave decomposes synthesis into four distinct, verifiable stages to enforce workflow dependencies and parameter provenance.

1. Tool Graph Synthesizer

Instead of inferring connections post-hoc via semantic similarity, ToolWeave employs an iterative curriculum to synthesize tools with built-in information flow:

• Seed Generation: Creates a minimal set of core tools.
• Entity Expansion: Uses structured knowledge from Wikipedia and Wikidata to expand coverage and infer business rules (e.g., making interval_days mandatory if recurring is true).
• Schema Enrichment: Adds depth via nested objects, enums, and defaults.
• Connection Discovery: Explicitly generates linking parameters (e.g., technician_id, equipment_id) to enable workflows, rather than relying on superficial name matching.
  This process yields a domain-specific tool graph where directed edges represent explicit data dependencies.

2. Structured Tool Graph Sampler

Given the synthesized graph, the framework samples coherent workflows aligned with meaningful user goals.

• Motif-Based Sampling: Instead of random walks, ToolWeave samples workflows following specific structural motifs: Linear (sequential dependencies), Fan-in/Fan-out (aggregating data before an action), and Conditional (branching based on API results).
• Goal Alignment: For each sampled workflow, a natural-language goal is synthesized. An LLM judge scores the compatibility between the workflow and the goal.
• Diversity Pruning: Maximum Marginal Relevance (MMR) is applied to linear workflows to prune similar goals, ensuring task diversity while retaining complex interaction patterns.

3. Structured Plan Generator

This stage shifts the reasoning burden from the dialogue synthesizer to an explicit planning phase to eliminate ambiguity.

• Tool Path Partitioning: The tool sequence is partitioned into contiguous groups (subgoals) representing coherent dialogue segments.
• Subgoal Synthesis: Each group is assigned a natural-language subgoal and realized as a sequence of interaction acts (user request, assistant clarification, tool invocation, etc.).
• Parameter Planning: Crucially, the planner explicitly traces parameter provenance by categorizing parameters as either user-provided or derivable (from prior tool outputs or history). This explicit dataflow planning prevents the synthesizer from hallucinating argument sources.

4. Multi-Agent Dialogue Synthesizer & Post-Processing

• Multi-Agent Synthesis: A plan-driven framework uses dedicated agents (memory, tool, and interaction agents) to instantiate the plan into natural language. The memory agent maintains state to ensure consistency across turns.
• Post-Processing: To ensure robustness, the synthesized dialogues undergo:
  • Linguistic Variation: Rephrasing to prevent overfitting.
  • Failure Injection: Introducing erroneous tool calls and recovery behaviors to simulate real-world API faults.
  • Schema Masking: Replacing function names with generic identifiers to force reasoning over rich schemas rather than surface labels.

Key Contributions

1. Structured Synthesis Pipeline: ToolWeave is the first framework to decompose tool-calling dialogue synthesis into verifiable stages that explicitly model workflow dependencies and parameter provenance.
2. Curriculum-Driven Tool Generation: By synthesizing tools with built-in data dependencies rather than inferring them, ToolWeave generates tool graphs with higher structural complexity and connectivity than those derived from real-world APIs (e.g., RapidAPI) or one-shot generation.
3. Significant Reduction in Hallucinations: The fine-grained planning stage reduces parameter hallucination rates from ~53% in prior approaches to nearly 20%.
4. Increased Multi-Step Interactions: The structured approach increases the prevalence of true multi-step tool interactions (where later calls consume earlier outputs) from ~7% in baseline datasets to ~45%.

Experimental Results

The authors evaluated models fine-tuned on ToolWeave data against those trained on ToolFlow and ToolDial across three public benchmarks: BFCL-V3, API Bank, and CONFETTI.

• Performance Gains: LLMs fine-tuned on ToolWeave consistently outperformed those trained on prior datasets. Notably, Llama-3.1-70B fine-tuned on ToolWeave achieved 39.75% accuracy on the BFCL-V3 multi-turn benchmark, compared to 23.50% when fine-tuned on SOTA ToolFlow data.
• Multi-Step Reasoning: The improvement was most pronounced in multi-step scenarios. ToolWeave-trained models achieved 3.6 times the multi-step accuracy of the base model, whereas baselines struggled with complex reasoning chains.
• Synthetic vs. Real APIs: Surprisingly, models trained on ToolWeave's synthetic APIs (generated via GPT-5-mini) outperformed those trained on real RapidAPI schemas (ToolWeave-R), suggesting that the explicit parameter dependency structure in synthetic data provides a cleaner training signal than noisy real-world APIs.
• Ablation Studies: Removing the fine-grained planner caused performance to collapse to 7.50%, confirming that turn-level planning is critical. Similarly, replacing the structured sampler with random walks degraded accuracy, validating the importance of motif-based workflow sampling.

Significance and Claims

The paper claims that the structure of the synthesis pipeline is a more critical driver of realistic multi-turn tool-calling capabilities than the scale or licensing status of the synthesis LLM. ToolWeave demonstrates that open-weight models (e.g., GPT-OSS-120B) can produce higher-quality training data than frontier-model-driven pipelines when guided by a structured, verifiable framework.

The authors conclude that by externalizing complex reasoning into structured stages, ToolWeave enables the reliable generation of high-quality tool-calling trajectories. This approach allows open-weight models to rival or surpass frontier models in tool-calling performance, provided the training data is synthesized with rigorous attention to workflow coherence and parameter provenance. The work suggests that future advancements in agentic AI depend less on simply scaling up data generation and more on the architectural rigor of the data synthesis process itself.