💬 NLP

From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agents

This paper introduces a reproducible, dataset-agnostic framework that converts text-based tool-calling benchmarks into audio evaluations without re-annotation, revealing significant model-dependent performance gaps between text and voice while validating open-source LLMs as effective, privacy-preserving judges.

Original authors: Md Tahmid Rahman Laskar, Xue-Yong Fu, Seyyed Saeed Sarfjoo, Quinten McNamara, Jonas Robertson, Shashi Bhushan TN

Published 2026-05-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Md Tahmid Rahman Laskar, Xue-Yong Fu, Seyyed Saeed Sarfjoo, Quinten McNamara, Jonas Robertson, Shashi Bhushan TN

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a smart assistant that can talk to you and also perform tasks, like checking your calendar or booking a flight. You have two main ways to build this:

The "Translator" Approach (Cascade): The assistant listens to your voice, hires a human-like translator to write down exactly what you said, and then a super-smart text-reader reads that note to decide what to do.
The "Super-Listener" Approach (Omni-Modal): The assistant listens to your voice directly and figures out what to do without ever writing it down first.

The big question is: Does the "Super-Listener" actually work better, or does the "Translator" approach still win?

This paper introduces a clever, reproducible "test track" to answer that question without having to build a whole new set of voice recordings from scratch.

The Problem: The "Text-Only" Trap

Most tests for these smart assistants are done using text. You type a command, and the computer answers. But in the real world, people speak.

Existing tests are like driving a car on a perfectly smooth, empty racetrack (text).
Real life is like driving on a bumpy, rainy road with traffic (voice).

Researchers wanted to know: If we take these perfect text tests and turn them into voice tests, how much does the performance drop?

The Solution: The "Voice-Translator" Framework

Instead of recording thousands of real people saying things (which is expensive and messy), the authors built a recipe to turn existing text tests into voice tests automatically.

Think of it like this:

They took a list of written instructions (the text benchmarks).
They fed them into high-tech "Text-to-Speech" machines (like a very advanced robot voice) to create audio files.
They added background noise (like a busy coffee shop or a car engine) to make it realistic.
Crucially: They kept the "answer key" (the gold labels) exactly the same.

This allows them to test the same question twice: once as text, and once as voice. This lets them measure exactly how much "noise" the voice adds to the system.

The Race: Who Won?

They tested seven different "Super-Listener" models (from companies like OpenAI, Google, and Alibaba) on two different types of tasks:

Confetti: A task where the assistant must pick the right tool and fill in the details correctly (e.g., "Book a flight to London on Tuesday").
When2Call: A task where the assistant must decide if it needs to use a tool at all, or if it should just chat.

The Results:

No One-Size-Fits-All: There was no single "best" model.
- On the "Confetti" task, Gemini-3.1-Flash-Live was the champion.
- On the "When2Call" task, GPT-Realtime-1.5 took the crown.
The "Voice Tax": Every model got slightly worse when switching from text to voice.
- Some models (like Qwen3) only lost a tiny bit of accuracy (like a runner slowing down slightly in the rain).
- Others (like GPT-Realtime-1.5) lost a significant amount of accuracy (like a runner slipping in the mud).
The "Translator" vs. "Super-Listener" Showdown:
- For some models, the "Translator" approach (Voice -> Text -> Action) was actually slightly better or equal to the "Super-Listener."
- For others, the "Super-Listener" was better.
- Conclusion: You can't just assume one architecture is always better. It depends entirely on which model you are using and what task you are doing.

Where Do They Fail?

The researchers looked at the mistakes and found a funny pattern:

The "Misheard Detail" Problem: Most of the time, the models got the big picture right (they knew they needed to book a flight), but they messed up the details (they booked it for the wrong day or the wrong city).
It's like a waiter who understands you want "dinner" but brings you "breakfast" because they misheard the order.

The "Judge" Problem

In the real world, companies often don't have an "answer key" to check if the AI did the right job. So, the paper tested using other AI models as "judges" to grade the performance.

They found that open-source judges (free, privacy-friendly models) with enough brainpower (at least 8 billion parameters) could grade the work just as well as expensive, proprietary judges. This is great news for companies worried about privacy.

The Takeaway

If you are building a voice assistant, don't just guess whether to use a "Super-Listener" or a "Translator."

Test it on your own data: Use this framework to convert your text logs into voice tests.
Check the "Voice Tax": See how much your specific model drops in performance when you add voice.
Watch out for ambiguity: If your users ask vague questions, the system will fail even more, regardless of whether it's text or voice.

In short: This paper gives companies a "stress test" to see if their voice assistants are ready for the real world, proving that the best choice depends on the specific model and the specific job, not just a general rule of thumb.

Technical Summary: From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agents

Problem Statement

While Large Language Models (LLMs) have advanced significantly in tool calling and function execution within text-based environments, prominent benchmarks (e.g., Confetti, When2Call, BFCL) remain text-only. This creates a gap for organizations deploying voice agents, who must choose between a cascade architecture (ASR $\rightarrow$ Text LLM) or an end-to-end omni-modal architecture (Speech $\rightarrow$ Tool Call). Existing audio benchmarks often require building new corpora from scratch with new tool schemas and gold labels, which is costly and limits applicability to proprietary enterprise data. Furthermore, the reliability of omni-modal models under realistic audio conditions and the specific performance degradation introduced by the speech modality remain insufficiently characterized.

Methodology

The authors propose a dataset-agnostic framework that converts existing text-based tool-calling benchmarks into controlled audio evaluations without re-annotating tool schemas or gold labels.

1. Conversion Pipeline

The framework utilizes off-the-shelf Text-to-Speech (TTS) models to generate paired text–audio instances:

TTS Models: Three models were used: Gemini-2.5-Flash-TTS, Gemini-2.5-Pro-TTS, and GPT-4o-Mini-TTS.
Speaker Variation: Distinct voices for male and female speakers were selected from each provider (e.g., Kore/Orus for Gemini, Coral/Ash for GPT).
Noise Injection: Environmental noise from the DEMAND dataset was injected at various Signal-to-Noise Ratios (SNR: 5, 10, 15, 20 dB) to simulate deployment conditions.
Preservation: The original tool schemas, gold labels, and evaluation protocols are strictly preserved from the source text datasets.

2. Evaluation Protocol

The framework evaluates seven omni-modal models across four providers (OpenAI, Google, Qwen, Microsoft) on two datasets:

Confetti: Focuses on function selection and parameter extraction accuracy (313 examples requiring tool calls).
When2Call: Focuses on the decision-making capability to determine whether a tool call is necessary (300 examples).

The evaluation compares three conditions:

Clean Text: Baseline performance using original text inputs.
Direct Voice: Performance using TTS-generated audio inputs processed by omni-modal models.
Cascade (ASR $\rightarrow$ Text): Audio transcribed by GPT-4o-Transcribe and passed to a text-only LLM.

3. Additional Analyses

Error Decomposition: Paired text–audio failures are categorized into decision errors, tool-selection errors, argument-schema errors, and argument-value errors.
Reference-Free Evaluation: LLM-as-a-Judge protocols (using GPT-5, Gemini-2.5-Pro, and open-source Qwen3 models) are tested against gold labels to assess viability for production data lacking annotations.
Stress Testing: Ambigu-based query reformulation tests were conducted to measure robustness to underspecified user requests.

Key Results

Model and Task Dependency

Performance is highly dependent on the specific model and task, rather than the architecture alone:

Confetti: Gemini-3.1-Flash-Live achieved the highest score (70.4%), followed by Qwen3-Omni (60.4%).
When2Call: GPT-Realtime-1.5 performed best (71.9%), while Gemini-3.1-Flash-Live dropped to 4th place (63.4%).
Text-to-Voice Gap: The performance degradation when moving from text to voice varies significantly by model. On Confetti, the gap ranges from 1.8 points (Qwen3-Omni) to 4.8 points (GPT-Realtime-1.5).

Architecture Comparison (Cascade vs. Omni)

Gemini-3.1-Flash-Live: The cascade architecture slightly outperformed direct voice (71.3% vs. 70.4%), suggesting ASR + strong text LLM can recover performance.
GPT-Realtime-1.5 & Qwen3-Omni: Direct voice outperformed the cascade by 0.4 and 1.5 points, respectively.
Conclusion: Neither architecture uniformly dominates; the optimal choice is model- and task-specific.

Error Analysis

Argument Values: The most common failure mode in audio-induced errors is argument-value errors (incorrect or incomplete field values), accounting for 57.2% of failures for Gemini-3.1-Flash-Live and 54.3% for GPT-Realtime-1.5. This indicates models often preserve the tool-call structure but misinterpret speech content.
Decision Errors: Significant decision errors (30–37%) were observed, indicating audio input alters high-level tool-use decisions (e.g., whether to call a tool at all).
When2Call Specifics: Models generally struggled more with "no-tool-call" cases than "tool-call" cases, highlighting a difficulty in knowing when not to act.

Evaluation and Scaling

LLM Judges: Reference-free LLM judges showed stable model rankings compared to reference-aware judges. Open-source Qwen3 judges with $\ge$ 8B parameters achieved >80% agreement with proprietary judges, supporting privacy-preserving evaluation.
Text Scaling: In text-only mode, accuracy scales with model size (e.g., Qwen3 rising from 26.6% at 0.6B to 70.7% at 32B). Smaller text models (e.g., 8B–14B) can serve as competitive cascade backends.
Ambiguity Stress Test: Ambiguous queries caused significant performance drops (e.g., Qwen3-Omni dropped 24.6 points), suggesting query formulation is a major source of variance independent of audio.

Significance and Claims

The paper claims to provide a verifiable and reproducible first-stage diagnostic for voice agent deployment. Its primary contributions are:

Benchmark Conversion: A method to convert trusted text benchmarks into audio evaluations without rebuilding gold labels or schemas, allowing enterprises to test proprietary data.
Diagnostic Signal: Evidence that the "text-to-voice" gap is not a fixed property of omni-modal inference but varies by model and task, necessitating specific diagnostics rather than general assumptions.
Error Isolation: The paired text–audio design enables counterfactual error analysis, revealing that audio failures often stem from argument-value misunderstandings rather than structural tool selection failures.
Practical Evaluation: Validation that reference-free LLM judges (including open-source options) can effectively evaluate tool calling in production settings where gold labels are unavailable.

The authors position this framework as a complement to purpose-built audio corpora, offering a controlled way to isolate modality-induced failures and inform architectural choices (cascade vs. omni) before deploying with natural speech. They acknowledge limitations, noting that TTS audio is a controlled proxy and does not fully capture the artifacts of spontaneous human conversation.