💬 NLP

Linguistic and Argument Diversity in Synthetic Data for Function-Calling Agents

This paper proposes a novel method for generating synthetic training data that optimizes linguistic and argument diversity without relying on hand-crafted rules, resulting in function-calling agents that achieve superior out-of-distribution performance and a 7.4% accuracy increase on the BFCL benchmark compared to state-of-the-art baselines.

Original authors: Dan Greenstein, Zohar Karnin, Chen Amiraz, Oren Somekh

Published 2026-01-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dan Greenstein, Zohar Karnin, Chen Amiraz, Oren Somekh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a very smart robot assistant how to use a giant toolbox filled with thousands of different tools (like weather apps, stock market trackers, or calendar schedulers). To teach the robot, you need to show it examples of people asking for help and the robot correctly picking the right tool.

The problem, according to this paper, is that the "textbooks" (datasets) used to train these robots are often too repetitive. They are like a cooking class where every student is only taught to make a cheese pizza with pepperoni. If a customer asks for a "vegan deep-dish with extra basil," the robot gets confused because it has never seen that specific request before.

Here is how the authors fixed this, using simple analogies:

1. The Problem: The "Echo Chamber" of Data

Previous methods of creating training data were good at making sure the robot knew many different tools existed. However, they failed at two important things:

Linguistic Diversity: They only taught the robot how to answer when people asked questions in the exact same stiff, formal way. They didn't teach it how to handle slang, casual chat, or weird sentence structures.
Argument Diversity: They only taught the robot about the most popular options. For example, if the tool is about stocks, the training data only ever mentioned "Apple" (AAPL) and "Microsoft" (MSFT). The robot never learned what to do if someone asked about a tiny, obscure company.

2. The Solution: The "Diversity Chef"

The authors built a new method to generate synthetic training data. Think of this method as a "Diversity Chef" who doesn't just follow a recipe book. Instead, this chef has a special rule: "Every time I make a new dish, it must taste different from the last 100 dishes I made."

No Rigid Rules: Unlike other methods that rely on a fixed list of "types of people" (e.g., "The Businessman," "The Student"), this chef doesn't need a pre-written list. It automatically figures out how to vary the requests.
The "Marginal Contribution" Trick: The chef looks at the pile of dishes already made. When considering a new dish, it asks: "If I add this new dish to the pile, does it make the whole collection more interesting?" If the answer is yes, it keeps the dish. If the dish is just a copy of something already there, it throws it away.

3. What They Actually Did

The researchers used this "Diversity Chef" to create a new set of training examples for function-calling agents. They focused on two main ingredients:

The Request (The Order): They generated user questions that varied wildly in how they were phrased (short, long, formal, slangy, confusing).
The Arguments (The Ingredients): They ensured the specific values used in the requests were varied. Instead of always using "New York" for a city, they used "Nairobi," "Reykjavik," and "Buenos Aires." Instead of always using "USD" for currency, they used "NOK," "RUB," and "XRP."

4. The Results: A Better Robot

They tested their new "Diversity Chef" data against other top-tier methods (like ToolAce and APIGen).

The Diversity Score: Their data was significantly more diverse. It wasn't just a little bit different; it was a whole new world of variety.
The "Out-of-Distribution" Test: This is the most important part. They trained a robot on their data and tested it on new questions it had never seen before (questions from other datasets).
- The Analogy: Imagine you trained a student only on math problems from 2020. Then you gave them a test with problems from 2025.
- The Outcome: The robot trained on the authors' diverse data performed much better on these new, unseen tests. It got about 7.4% more questions right on a major benchmark (BFCL) compared to robots trained on the older, less diverse methods.

Summary

The paper claims that by using a smart, automated process to ensure training data is linguistically varied (different ways of speaking) and argumentatively varied (different specific values), you can build a much more robust AI agent. This agent doesn't just memorize specific examples; it learns to handle the messy, unpredictable reality of how real humans actually talk and what they actually ask for.

What they did NOT claim:

They did not claim this works for multi-turn conversations (long back-and-forth chats); they focused strictly on single-turn requests.
They did not claim this works for languages other than English.
They did not claim this is a "cure-all" for all AI problems, but specifically for the task of teaching agents to call tools correctly.

Technical Summary: Linguistic and Argument Diversity in Synthetic Data for Function-Calling Agents

Problem Statement

The development of function-calling agents, which enable Large Language Models (LLMs) to interact with external APIs and tools, is currently hindered by a scarcity of high-quality, diverse training data. While prior research has addressed the need for diversity in the functions themselves (covering various domains), the invocation structures (single vs. multi-turn), and the response types (call vs. no-call), two critical dimensions remain underexplored:

Linguistic Diversity: The variation in the shape, style, and semantic expression of user requests. Existing datasets often rely on shallow variations or fixed prompt templates, failing to capture the full spectrum of how users might phrase identical intents.
Argument Diversity: The variation in parameter values (e.g., specific city names, stock tickers, dates). Generated datasets tend to concentrate on "head" values (e.g., "New York," "AAPL"), leaving models potentially unprepared for less common or "tail" values, which limits robustness in out-of-distribution (OOD) scenarios.

Methodology

The authors propose a novel data generation pipeline that optimizes for diversity without relying on hand-crafted rules, predefined taxonomies, or static persona pools. The core of their approach is a greedy diverse generation procedure that iteratively selects items to maximize a marginal diversity contribution metric.

Core Components

Diverse Generation Framework:
- The system operates in rounds, generating candidate items and selecting the top $b$ candidates based on a diversity score $D(c; E)$ , defined as the marginal increase in a set-level diversity metric $\tilde{D}$ when adding candidate $c$ to the existing set $E$ .
- The process uses an LLM to generate candidates based on instructions $I$ and existing examples $E$ .
- Crucially, the instructions are updated in subsequent rounds based on the scores of previous candidates and a filtering process $F$ that eliminates invalid or incorrect items. This creates a feedback loop that steers generation toward underrepresented regions of the data space.
Argument Value Generation:
- Preprocessing: Parameters across different APIs are clustered based on semantic similarity (using embeddings) to identify groups (e.g., "city name," "stock ticker").
- Optimization: For numerical and string parameters, the system employs a "virtual augmentation" strategy. The LLM generates a large pool of candidates (e.g., 25), from which a subset is selected to maximize Cluster Entropy. This metric measures the entropy of the distribution of values across semantic clusters, encouraging the selection of tail values rather than frequent head values.
- Consistency: For sequential API calls, parameters are generated backward from the last API to ensure logical cohesion, while parallel calls ensure shared context (e.g., same location) across APIs.
Query (Request) Generation:
- Given a target function and argument values, the system generates natural language queries.
- Diversity is optimized using a composite metric derived via Reciprocal Rank Fusion (RRF). This fuses rankings from multiple independent metrics:
  - Lexical: Type-Token Ratio (TTR), Compression Ratio.
  - Syntactic: Parse tree entropy, variance in query length.
  - Semantic: Paraphrase variety (embedding similarity), Chamfer distance, Vendi score.
- An LLM judge validates that the generated query logically justifies the specific API call and parameters.
Distractor Selection:
- To create realistic training examples, the system retrieves "hard negative" distractors (irrelevant APIs) that are semantically close to the query but not the correct answer. This is achieved via embedding-based retrieval followed by an LLM-based plausibility filter and an "elbow method" to determine the cutoff for relevance scores.

Key Contributions

A Novel Diversity Component: A general-purpose, rule-free mechanism for generating diverse items by optimizing marginal diversity metrics. This component is adaptable to new domains without manual reconfiguration.
Synthetic Data Generation Technique: The application of this component to function-calling data, specifically targeting linguistic variation in queries and argument value diversity in parameters.
Comprehensive Analysis: A thorough evaluation demonstrating that increasing these specific diversity dimensions improves model performance, particularly in OOD settings.

Results

The authors evaluated their method against State-of-the-Art (SoTA) baselines, specifically ToolAce and APIGen, as well as the BFCL benchmark.

Intrinsic Evaluation (Data Quality)

Argument Diversity: The proposed method achieved significantly higher NCD Diversity and Cluster Entropy scores compared to ToolAce and APIGen. For instance, in currency arguments, while baselines were dominated by "USD," the proposed method produced a balanced distribution across various currencies.
Linguistic Diversity: The generated dataset outperformed baselines on nearly all linguistic metrics (Simpson's Index, Tree Edit Distance, Semantic Spread, Vendi Score) with statistical significance. The only exception was query length variance, where baselines had outliers.
Correctness: Manual inspection and automated judging confirmed that the increased diversity did not compromise data correctness; the generated examples remained realistic and valid.

Extrinsic Evaluation (Model Performance)

The authors fine-tuned Llama-3.1-8B-Instruct using their dataset and compared it against models fine-tuned on ToolAce and APIGen data.

Out-of-Distribution (OOD) Performance: Models trained on the proposed dataset showed superior generalization when tested on datasets they were not trained on.
- On the APIGen test set, the proposed model outperformed the ToolAce-trained model by 4.2 accuracy points.
- On the ToolAce test set, the proposed model outperformed the APIGen-trained model by 3.8 accuracy points.
BFCL Benchmark: When evaluated on the Berkeley Function-Calling Leaderboard (non-live subset), the model trained on the proposed dataset achieved 85.2% accuracy, significantly outperforming the ToolAce (77.8%) and APIGen (77.1%) counterparts. This represents a ~7.4% increase in accuracy over the baselines.

Significance and Claims

The paper claims that the lack of linguistic and argument diversity in current synthetic datasets is a bottleneck for the robustness of function-calling agents. By directly optimizing for these dimensions using a general-purpose, metric-driven approach, the authors demonstrate that:

Diversity drives Generalization: Increased diversity in training data leads to measurable improvements in OOD performance, suggesting that models trained on diverse data are better at handling unseen user phrasings and rare parameter values.
Automation over Manual Curation: The proposed method achieves superior diversity without the need for manual rule-writing or fixed persona definitions, making it scalable and adaptable to new domains.
Practical Impact: The technique provides a pathway to generate high-quality training data that bridges the gap between synthetic benchmarks and real-world agent performance, as evidenced by the significant gains on the BFCL benchmark.

The authors remain modest regarding limitations, noting that their current analysis is restricted to English, the generation process relies on strong (and thus costly) LLMs for filtering, and the study focuses on single-turn interactions, leaving multi-turn diversity as a direction for future work.