Technical Summary: Simulating Complex Multi-Turn Tool Calling Interactions in Stateless Execution Environments

Problem Statement

Synthetic data is a critical resource for tuning smaller, cost-effective Large Language Models (LLMs) to handle complex multi-turn tool calling. However, existing frameworks for generating such data frequently assume the availability of a stateful execution environment. In these environments, the validity of an interaction is determined by whether the execution state matches a pre-specified objective.

This assumption fails in many real-world scenarios, particularly in enterprise settings where data security prohibits privileged access to live backends, or when tool specifications are synthesized from multiple sources without a unified state. Furthermore, many top-tier open language models are trained on synthetic, LLM-generated tools that can only be simulated. Current methods struggle to generate meaningfully complex, multi-turn interactions with implicit tool calls (where the user request does not explicitly specify every necessary tool) without access to a real, stateful backend.

Methodology: DiGiT-TC

The authors introduce DiGiT-TC (Data Generation and Transformation for Tool-Calling), a framework designed to produce synthetic multi-turn tool calling conversations that mimic the complexity of stateful exploration, despite operating in a stateless environment. The core innovation is a "flip" in the generation process: instead of generating a user request and then deriving tool calls, DiGiT-TC first generates a sequence of tool calls and then constructs a corresponding user request.

The framework operates through the following stages:

1. Initialization: A conversation is initialized with an empty message list, a sampled set of tools, and a user persona. The persona is augmented with specific details (e.g., location, profession) relevant to the sampled tools.
2. Plan Distillation:
   • An LLM generates a raw sequence of tool calls ($S$) based on the persona and tools. These calls include inter-tool references (e.g., using Pythonic dot notation like $1.output to pass data between calls).
   • The sequence is treated as a Directed Acyclic Graph (DAG). The framework distills this into a more concise subsequence by selecting the largest connected component, filtering out redundant or disconnected tool calls.
3. Request Generation (Implicit Tool Call Selection):
   • The distilled sequence $S$ is partitioned into two sets: Explicit ($E$) and Implicit ($I$).
   • An algorithm randomly selects a target size for the implicit set. It iteratively removes elements from $S$ that are not dependencies of other elements, ensuring the remaining implicit set contains tool calls that are necessary for the explicit set but not directly mentioned by the user.
   • The LLM generates a user request ($r$) that explicitly targets the sequence $E$ while avoiding any direct mention of the tools in $I$.
   • Validation: To ensure the user request is not a trivial transcription of the tool sequence, the framework calculates Kendall's $\tau_b$ correlation coefficient between the rank of arguments in the tool calls and their appearance order in the user request. If the correlation is too high (indicating a linear, explicit mapping), the example is filtered out.
4. Back Translation:
   • To mitigate noise and ensure the generated user request actually leads to the intended tool sequence, the framework performs a back-translation step.
   • An LLM is prompted to predict a new tool sequence ($S'$) based only on the generated user request ($r$) and the available tools.
   • The system validates that the "leaves" (final outputs) of the original sequence $S$ are represented in $S'$. If the condition leaves(S) \ leaves(S') = ∅ fails, the example is discarded.
5. Termination and Augmentation:
   • Execution & Summarization: The tool calls are simulated (executed via an LLM) to substitute variable values and generate a summary of the results.
   • Error Augmentation: The framework introduces complexity by simulating two types of errors:
     • Human Errors: Unsolvable or incomplete user requests requiring clarification.
     • Tool Execution Errors: Simulated incorrect tool calls followed by a hint (generated by an LLM) guiding the model toward the correct solution.

Key Contributions

1. Stateless Generation of Implicit Calls: DiGiT-TC is the first framework to systematically generate complex multi-turn interactions with implicit tool calls without requiring a stateful execution backend.
2. Selective Generation Strategy: By flipping the generation order (Tool Calls $\to$ User Request), the method gains fine-grained control over which calls are explicit and which are implicit.
3. Noise Mitigation via Back Translation: The introduction of a back-translation step ensures that the generated user requests are robust and faithfully map to the intended tool sequences, filtering out hallucinations or misalignments.
4. Open-Source Ecosystem: The authors fully open-source the data generation framework, the prompts, and the resulting dataset (5,000 data points), all produced using GPT-OSS-120B, a license-friendly, open-weights teacher model.

Experimental Results

The authors evaluated DiGiT-TC on three standard benchmarks: BFCL (Berkeley Function Calling Leaderboard), $\tau$-bench, and ACEBench.

• Performance Gains: Fine-tuning base models (e.g., Granite-4.1-3B, Llama3.1-8B-Inst) with DiGiT-TC data resulted in significant performance improvements (ranging from 6 to 25 percentage points) over the base models.
• Benchmark Comparison:
  • On the BFCL Multi-Turn category, DiGiT-TC (using Llama3.1-8B-Inst) achieved 35.5% accuracy, outperforming other open-data approaches like ToolWeave (25.0%) and Magnet (26.5%), and approaching the performance of models trained on frontier data (e.g., ToolACE-MT at 40.3%).
  • The method demonstrated strong generalizability across $\tau$-bench and ACEBench, consistently outperforming prior approaches trained exclusively on open-source data.
• Ablation Studies:
  • Removing Implicit Tool Calls caused a 4% drop in performance, confirming their necessity for complex reasoning.
  • Removing Back Translation caused a 3% drop, highlighting its role in reducing noise.
  • Using a smaller teacher model (GPT-OSS-20B) resulted in a ~2% drop, suggesting that while larger models help, the method remains effective with smaller, accessible models.

Significance and Claims

The paper claims that DiGiT-TC addresses a critical gap in the development of agentic AI: the ability to train smaller models on complex, multi-turn tool calling data without the privacy risks and costs associated with accessing real, stateful backends.

• Privacy and Scalability: By relying solely on tool specifications and open-weights models, the approach avoids the data security risks of interfacing with enterprise backends.
• Cost-Effectiveness: The authors note that using GPT-OSS-120B is significantly cheaper than using frontier models like GPT-4o (approx. $0.015 per example vs. significantly higher costs for GPT-4), making high-quality synthetic data generation accessible to a broader range of practitioners.
• State-of-the-Art for Open Models: The work sets a new state-of-the-art for multi-turn tool calling performance among models trained exclusively on data produced by open-source, open-weights teacher models.

The authors modestly acknowledge that for tasks where a full simulation of a system's state is available, methods leveraging that state might still be superior. However, DiGiT-TC provides a robust, repeatable process for generating high-quality training data in the absence of such environments.