💬 NLP

SynAE: A Framework for Measuring the Quality of Synthetic Data for Tool-Calling Agent Evaluations

The paper introduces SynAE, a multi-axis evaluation framework designed to assess the validity, fidelity, and diversity of synthetic datasets for tool-calling agents by analyzing task instructions, tool calls, outputs, and downstream performance, demonstrating that no single metric suffices to characterize synthetic data quality.

Original authors: Shuaiqi Wang, Aadyaa Maddi, Zinan Lin, Giulia Fanti

Published 2026-05-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shuaiqi Wang, Aadyaa Maddi, Zinan Lin, Giulia Fanti

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to perfect a new recipe. Usually, you taste your dish using real ingredients from the market. But sometimes, you can't use the real market (maybe it's too expensive, or the ingredients are secret). So, you decide to cook with synthetic ingredients—fake vegetables and artificial meat made in a lab.

The problem? You don't know if your fake ingredients will actually taste like the real thing, or if they'll make the dish fall apart.

This paper introduces SynAE, a "taste-tester" framework designed specifically to check the quality of these synthetic ingredients before you serve them to your customers (in this case, AI agents).

Here is how SynAE works, broken down into simple concepts:

1. The Problem: The "Fake Data" Blind Spot

AI agents (smart programs that can use tools like search engines or calculators) need to be tested. Usually, developers test them on real data (records of real people asking for help). But real data is often private, sensitive, or just too small to test everything.

So, developers create synthetic data—fake conversations and tasks generated by computers to mimic real life.

The Risk: Just because the fake data looks like real data doesn't mean it acts like real data. It might look perfect but fail when the AI tries to use it.
The Gap: Until now, there was no standard "ruler" to measure exactly how good or bad this fake data was.

2. The Solution: SynAE (The Quality Control Inspector)

SynAE is a framework that compares the Real Dataset (the gold standard) against the Synthetic Dataset (the fake one). It doesn't just say "Good" or "Bad"; it measures three specific qualities:

A. Validity: "Does it actually work?"

The Analogy: Imagine a fake recipe that says "Add 5 cups of sugar." If you follow it, your cake burns. That's invalid.
What SynAE checks: It asks, "If an AI follows these fake instructions, does it actually finish the task?" It checks if the tool calls (like clicking a button) and the final answers make sense and solve the problem. If the instructions are confusing or lead to a dead end, SynAE flags it as "Invalid."

B. Fidelity: "Does it feel like the real thing?"

The Analogy: Imagine a wax fruit. It looks like an apple, but if you bite it, it's hard and tasteless. It lacks the "fidelity" (faithfulness) of a real apple.
What SynAE checks: It compares the fake data to the real data to see how similar they are.
- Structure: Do the fake conversations have the same back-and-forth rhythm as real ones?
- Patterns: Do the fake agents use tools in the same order and frequency as real agents?
- Content: Are the fake questions and answers semantically close to the real ones?
- If the fake data is too different from the real world, it has low fidelity.

C. Diversity: "Is it boring or interesting?"

The Analogy: Imagine a playlist where the same song plays 100 times. It's not diverse. Now imagine a playlist with 100 different genres. That's diverse.
What SynAE checks: It measures if the synthetic data covers a wide variety of scenarios. If the fake data just repeats the same few types of questions over and over, it has low diversity. This is bad because the AI won't learn how to handle new, weird situations.

3. How They Tested It

The authors didn't just talk about it; they built a "stress test" lab. They took real datasets and intentionally broke them in specific ways to see if SynAE could catch the errors:

The "Blank Filling" Test: They took real instructions and hid random words, forcing an AI to guess the rest. As they hid more words, the data got worse. SynAE correctly detected that the fidelity dropped (it didn't look like the real thing anymore) but diversity went up (the guesses were all over the place).
The "Copy-Paste" Test: They took a few real examples and copied them 100 times. SynAE correctly flagged that diversity crashed (it was all the same) even though the fidelity stayed high (it still looked like the real thing).
The "Broken Tool" Test: They kept the instructions but gave the AI the wrong tools to use. SynAE caught that the validity was gone because the tasks couldn't be completed.

4. The Big Discovery

The most important finding is that no single number tells the whole story.

You can have data that is very valid (it works) but has low diversity (it's boring).
You can have data that is very diverse (it's exciting) but has low fidelity (it doesn't look like the real world).
You can have data that looks perfect (high fidelity) but is broken (low validity).

Conclusion

SynAE is like a multi-axis dashboard for AI developers. Instead of just guessing if their fake data is good, they can use SynAE to see exactly where it is failing. Is it too repetitive? Is it too different from reality? Is it actually broken?

The paper concludes that before you trust synthetic data to test your AI agents, you need this kind of detailed, multi-dimensional checkup to ensure you aren't training your AI on a "wax fruit" that looks real but has no flavor.

Technical Summary: SynAE – A Framework for Measuring the Quality of Synthetic Data for Tool-Calling Agent Evaluations

1. Problem Statement

Tool-calling agents are increasingly evaluated using static datasets of execution traces (comprising user inputs, agent responses, tool calls, and final outputs). However, practitioners often face significant barriers in utilizing real-world production data for testing due to privacy restrictions, proprietary concerns, or data sparsity. Consequently, there is a growing reliance on synthetic datasets to replace or augment real data for evaluation purposes.

The core problem identified is the lack of systematic methods to quantify the relationship between these synthetic datasets and the real data they aim to replicate. Current literature on testing agents with synthetic data provides almost no quantitative methods for evaluating the quality of such data, leaving operators with little visibility into evaluation gaps. Existing evaluation metrics typically focus on whether an agent performs correctly on a benchmark, rather than evaluating the benchmark itself for validity, fidelity, and diversity.

2. Methodology: The SynAE Framework

The authors introduce SynAE, a comprehensive evaluation framework designed to assess how well synthetic trajectories replicate and augment the characteristics of real data trajectories. SynAE operates by taking a real baseline dataset ( $D$ ), a synthetic dataset ( $D'$ ), and optional metric configurations (including prior knowledge and specific LLM agents) as inputs.

The framework evaluates synthetic data across three primary properties, measured through four distinct evaluation aspects:

Task Instructions and Responses
Tool Calls
Final Outputs
Downstream Evaluation

2.1 Evaluation Pillars

SynAE divides its metrics into three pillars:

A. Validity
Validity assesses whether synthetic tool calls and outputs successfully fulfill the given instructions.

Metric: Validity Rate (VR).
Method: By default, an LLM-as-a-judge (Mistral-7B-Instruct) is used to determine if tool calls or outputs accomplish the task. Rule-based checkers are used where available. A sample is considered valid only if the tool call or output fully satisfies the instruction (partial correctness is rejected).

B. Fidelity
Fidelity measures the similarity between synthetic and real data, crucial when synthetic data replaces real data due to privacy constraints.

Instructions & Responses:
- Key Node Dependency (KND): Measures semantic dependencies between instructions and responses via embedding cosine similarities.
- Attribute Match (AM): Computes distributional distances (Wasserstein-2 for numerical, Total Variation for categorical) over attributes like turn counts and token lengths.
- KNN-Precision/Recall & FID: Measure semantic quality and coverage using embedding distances.
Tool Calls:
- Tool Usage Match (TUM): Compares overall tool-usage distributions.
- Tool Call Number Match (TCNM): Compares distributions of tool-call counts per sample.
- k-Step Tool Planning Match: Measures similarity in conditional distributions of the next tool given previous $k-1$ tool calls.
Outputs: Uses KNN-Precision, KNN-Recall, and FID relative to the real dataset.
Downstream Evaluation:
- Task Difficulty Difference (TDD): Measures the absolute difference in task-completion performance between real and synthetic tasks across multiple agents.
- Ranking Divergence (RD): Computes Spearman rank correlation to determine if agent performance rankings are preserved.

C. Diversity
Diversity is critical when synthetic data is used to improve coverage (e.g., adding outliers). These metrics are reference-free (do not require real data).

Vendi Score (Vendi): Computes diversity based on the exponential entropy of eigenvalues of a similarity matrix (cosine similarity for text, Levenshtein distance for tool call sequences).
Attribute Diversity (AD): Measures the entropy of user-specified attribute distributions (e.g., attraction types, cities).

3. Experimental Setup

The authors evaluated SynAE on three recent agent benchmarks treated as "real" datasets:

T1: Multi-turn attraction recommendation (225 samples).
BFCL: Multi-turn function calling across domains like file operations and travel (200 samples).
ACP: Planning domains including transportation and robot actions (260 samples).

Synthetic Data Generation Methods:
To test SynAE's sensitivity, the authors constructed synthetic datasets using:

Controlled Modifications:
- Blank Filling: Randomly masking tokens in instructions to degrade fidelity.
- Oversampling: Duplicating samples to reduce diversity.
- In-Context Generation: Varying the number and randomness of in-context examples.
- Invalidation: Replacing valid tool calls/outputs with incorrect ones to degrade validity.
Industry Standard: NVIDIA NeMo Data Designer using various backend models (GPT-4o-mini, Llama3.1, Nemotron, Mistral) and temperatures.

4. Key Results

Fine-Grained Detection: SynAE successfully captures fine-grained variations in validity, fidelity, and diversity. For instance, increasing the masking probability in "Blank Filling" degraded fidelity metrics (KNN-Precision) while increasing diversity (Vendi Score). Conversely, "Oversampling" reduced diversity and coverage (KNN-Recall) while preserving some fidelity.
Invalidation Sensitivity: The framework detected a monotonic decrease in Validity Rate as the ratio of invalid samples increased, confirming its ability to spot unusable data.
Trade-offs: Experiments revealed that no single metric category fully characterizes synthetic data. For example, increasing the number of in-context examples ( $k$ ) did not uniformly improve all fidelity metrics; it improved KNN-Precision in some cases but lowered KNN-Recall.
Downstream Impact: The framework demonstrated that synthetic datasets with high fidelity in instructions might still fail in downstream evaluation if tool-planning patterns diverge, highlighting the necessity of multi-axis evaluation.
Iterative Refinement Case Study: In a simulated workflow, SynAE helped a practitioner diagnose a skewed dataset (low diversity), attempt a fix via "Relabeling" (which improved diversity but broke validity), and finally achieve a balanced result using NVIDIA NeMo with a high-capacity backend (GPT-4o-mini), which preserved validity and fidelity while improving diversity.

5. Significance and Claims

The paper claims that no single metric is sufficient to fully characterize synthetic data quality for agent testing. The significance of SynAE lies in its ability to provide a multi-axis evaluation that:

Quantifies the gap between synthetic and real data across semantic, structural, and statistical dimensions.
Detects specific failure modes (e.g., hallucinated tool names, lack of diversity, or degraded planning patterns) that single-metric approaches might miss.
Supports iterative refinement, allowing practitioners to diagnose bottlenecks and adjust generation strategies (e.g., model selection, temperature, or augmentation techniques) based on quantitative feedback.

The authors position SynAE as a plug-in component for agentic workflows, enabling operators to automatically evaluate the quality of synthetic benchmark datasets before deployment. They acknowledge limitations, noting the current focus is on multi-turn tool-calling settings, and extending the framework to interactive environments or multi-agent coordination is a direction for future work.