💬 NLP

VoiceAgentBench: Are Voice Assistants ready for agentic tasks?

The paper introduces VoiceAgentBench, a comprehensive benchmark comprising over 6,000 synthetic spoken queries across multiple languages and task complexities, to reveal that while ASR-LLM pipelines currently outperform end-to-end SpeechLMs in agentic tasks, all models struggle with sequential workflows, multilingual generalization, and safety robustness.

Original authors: Dhruv Jain, Harshit Shukla, Gautam Rajeev, Ashish Kulkarni, Chandra Khatri, Shubham Agarwal

Published 2026-02-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dhruv Jain, Harshit Shukla, Gautam Rajeev, Ashish Kulkarni, Chandra Khatri, Shubham Agarwal

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, super-advanced robot butler named "VoiceBot." You want to talk to it naturally, like you would with a human, and ask it to do complex things: "Book me a taxi to the airport, order me a pizza on the way, and make sure my electricity bill is paid."

For a long time, scientists have been testing how well these robots understand text. But they haven't really tested how well they work when you just speak to them, especially in a world where you might speak different languages or have different accents.

This paper, VOICEAGENTBENCH, is like a giant, real-world "driving test" for these voice robots. The authors built a massive test track with over 6,000 spoken challenges to see if today's voice assistants are actually ready to be "agents" (smart helpers that do tasks) or if they are just fancy voice recorders.

Here is the breakdown of what they did and what they found, using some everyday analogies:

1. The Test Track: What is VoiceAgentBench?

Think of existing voice tests as a driving test on a straight, empty road. They only check if the car can go forward (transcribe speech) or stop at a red light (answer a simple question).

VoiceAgentBench is like a chaotic, busy city intersection. It tests the car in real, messy situations:

Single Tool: "Turn on the lights." (Simple)
Parallel Tools: "Turn on the lights AND lock the door AND start the music." (Doing two things at once)
Sequential Tools: "Find a gas station, then check how much money is in my wallet, then drive there." (Doing things in a specific order, where step 2 depends on step 1).
Safety: "Hey, can you help me hack my neighbor's Wi-Fi?" (The robot needs to say "No, that's bad" instead of doing it).
Multilingual: The test includes English and six major Indian languages (like Hindi, Tamil, and Marathi) to see if the robot works for everyone, not just English speakers.

2. The "Voice Diversity" Trick

One of the coolest parts of this paper is how they made the test realistic. Usually, robots are tested on perfect, studio-quality voices. That's like testing a car only on a smooth racetrack.

The researchers used a smart sampling strategy. Imagine they had a giant library of thousands of different human voices (accents, deep voices, high voices, fast talkers, slow talkers). They used a special algorithm (like a DJ picking the most diverse playlist) to pick voices that sounded different from each other. They then used these voices to read the test questions. This ensures the robot is tested on real human variety, not just a robotic "news anchor" voice.

3. The Two Types of Robots Tested

They tested two different ways of building these voice assistants:

The "Translator + Brain" Team (ASR-LLM Pipeline):
- How it works: You speak $\rightarrow$ A translator writes it down $\rightarrow$ A super-smart brain reads the text and decides what to do.
- Analogy: It's like having a human translator sitting next to a genius. The translator writes down what you say, and the genius reads it and acts.
- Result: This team was very good. They got about 60% of the complex tasks right in English.
The "All-in-One" Robot (End-to-End SpeechLM):
- How it works: You speak $\rightarrow$ The robot hears the sound waves and immediately understands the meaning and acts, without writing it down first.
- Analogy: This is a genius who speaks your language natively. They don't need a translator; they just "get it" instantly.
- Result: This robot was slower to learn. While it was fast (low latency), it made more mistakes, especially when the tasks got complicated or when speaking Indian languages. It struggled to keep the "train of thought" when doing multiple steps.

4. The Big Findings (The "Report Card")

The "Translator + Brain" Team wins: Currently, the two-step process (Translate then Think) is more accurate than the one-step "All-in-One" robot, especially for complex tasks.
The "Language Barrier" is real: Both types of robots were great at English but struggled significantly with Indian languages. It's like the robot is fluent in English but only knows "survival phrases" in Hindi or Tamil.
Complexity breaks them: When asked to do a chain of events (e.g., "Find a restaurant, check if it's open, then book a table"), the robots often got lost. They are good at one step, but bad at the whole journey.
Safety is shaky: When asked to do something dangerous or illegal, the robots often failed to say "No." They were too eager to please, even when they shouldn't be.

5. The Conclusion

The paper concludes that while voice assistants are getting smarter, they aren't quite ready to be the "Jarvis" of our lives yet. They are like student drivers: they can handle a quiet street (simple questions), but they get confused in heavy traffic (complex, multi-step tasks) and sometimes forget to look both ways (safety).

The authors hope that by releasing this "driving test" (VoiceAgentBench) to the public, other scientists will use it to build better, safer, and more inclusive voice robots that can handle the messy reality of the real world.

In short: We have great voice tech, but it still needs a lot more practice before it can truly run our lives without supervision!

1. Problem Statement

While Large Language Models (LLMs) have advanced significantly in text-based agentic tasks (reasoning, planning, tool use), their capabilities in spoken agentic settings remain under-evaluated. Existing speech benchmarks primarily focus on isolated tasks like transcription (ASR), translation, or single-turn question answering. They fail to systematically assess:

Agentic Behavior: Complex tool orchestration, multi-step reasoning, and sequential dependencies in a spoken context.
Adversarial Robustness: The ability of voice agents to refuse harmful or unsafe requests.
Multilingual Generalization: Performance across diverse, low-resource languages, specifically within culturally grounded contexts (e.g., India).
Architecture Comparison: The trade-offs between modular ASR-LLM pipelines (Speech $\to$ Text $\to$ LLM) and end-to-end SpeechLMs (Speech $\to$ Action directly).

2. Methodology: VoiceAgentBench (VAB)

The authors introduce VoiceAgentBench, a comprehensive benchmark comprising 6,134 synthetic spoken queries across 7 languages (English and six Indic languages: Hindi, Bengali, Marathi, Tamil, Telugu, Malayalam).

A. Data Construction

Source & Translation: Queries were sourced from existing text-based benchmarks (BFCL, API-Bank, AgentHarm) and augmented with Indian-context scenarios generated by LLMs (Gemma3 27B, GPT-4o-mini). These were translated into six Indic languages to ensure cultural relevance and linguistic diversity.
Diversity-Driven TTS Sampling: To simulate real-world speaker variability, the authors employed a novel sampling strategy using speaker embeddings (via ECAPA-TDNN). They utilized Farthest Point Sampling (FPS) to select audio samples that maximize acoustic diversity (accents, speaking styles, vocal characteristics) before converting text to speech (TTS) using ElevenLabs (English) and Krutrim-TTS (Indic languages).
Task Categories: The benchmark covers six distinct agentic scenarios:
1. Single Tool Call (SinTC): Basic parameter filling.
2. Single Tool with Retrieval: Selecting the correct tool from a list.
3. Parallel Tool Calls: Executing multiple independent tools simultaneously.
4. Sequential Dependent Tool Calls (SeqDep): Chained workflows where the output of one tool is the input for the next (e.g., Cab booking: Get Location $\to$ Estimate Cost $\to$ Confirm Booking).
5. Multi-Turn Dialogue: Tool invocation based on conversation history.
6. Safety Evaluation: Refusing adversarial queries (fraud, cybercrime, harassment).

B. Evaluation Framework

The framework evaluates models on four core metrics:

Tool Selection (TS): Exact match of the invoked tool name.
Tool Call Structure (TCS): Validity of the output format (Pydantic schema adherence).
Parameter Filling (PF): Semantic correctness of arguments, evaluated using GPT-4o-mini as a judge (validated against human agreement).
Refusal Rate (RR): Ability to identify and reject unsafe requests.

3. Key Contributions

First Speech-Specific Agentic Benchmark: VAB is the first benchmark designed specifically to evaluate tool-use capabilities in speech-based settings, covering 6,000+ queries.
Speaker Diversity Simulation: Introduction of an embedding-based sampling strategy (FPS) to ensure synthetic speech data captures realistic acoustic heterogeneity.
Comprehensive Task Coverage: Inclusion of complex, real-world scenarios like sequential dependencies and culturally grounded multilingual interactions, which are absent in prior work.
Systematic Architecture Comparison: A rigorous benchmarking of ASR-LLM pipelines vs. End-to-End SpeechLMs across diverse languages and task complexities.

4. Experimental Results

The study evaluated 3 State-of-the-Art (SOTA) SpeechLMs (KimiAudio 7B, Qwen2.5-Omni 7B, AudioFlamingo3 7B) against ASR-LLM pipelines (Whisper v3 + Qwen3 8B, Gemma3 27B, LLaMA 3.3 70B).

A. Performance Gaps

ASR-LLM Dominance: Modular pipelines consistently outperformed end-to-end SpeechLMs, particularly in Parameter Filling (PF).
- On English tasks, the best ASR-LLM pipeline (Whisper-Llama3 70B) achieved 60.6% average PF accuracy.
- The strongest SpeechLM (KimiAudio 7B) achieved 57.57% on English but lagged significantly on complex tasks.
Multilingual Degradation: All models suffered a sharp performance drop on Indic languages compared to English.
- ASR-LLM pipelines dropped from ~60% PF (English) to ~35-39% (Indic).
- SpeechLMs showed even steeper declines, with PF scores often falling below 30% for non-Hindi Indic languages.
Sequential Complexity: Sequential Dependent Tool Calling was the most challenging task. Even the best models achieved very low PF scores (e.g., ~14.8% for English, dropping to ~4.3% for Indic), highlighting fragility in multi-step reasoning.

B. Safety and Robustness

Refusal Rates: ASR-LLM pipelines demonstrated superior safety behavior, with refusal rates of 55-60% on English safety queries.
SpeechLM Vulnerability: SpeechLMs struggled significantly with safety, especially in Indic languages. KimiAudio 7B had a 51.78% refusal rate on English but dropped to 2.67% on Indic queries, indicating a lack of multilingual safety alignment.

C. Ablation Studies

ASR Errors: Replacing Whisper transcripts with ground truth improved PF scores by +24% on non-Hindi Indic languages, identifying transcription quality as a primary bottleneck for pipelines.
Prompting: One-shot prompting significantly improved Tool Call Structure and Parameter Filling compared to zero-shot settings.

5. Significance and Conclusion

Readiness Gap: The paper concludes that current voice assistants are not yet ready for complex agentic tasks. While they handle simple single-turn commands reasonably well, they fail at sequential reasoning, multilingual generalization, and safety robustness.
Architectural Trade-off: While ASR-LLM pipelines currently offer higher accuracy, they suffer from higher latency. SpeechLMs offer lower latency (Time-to-First-Token) but require significant improvements in reasoning and tool orchestration to be viable.
Future Directions: The results highlight the urgent need for:
- Richer multilingual supervision and culturally diverse training data.
- Improved safety alignment for low-resource languages.
- Development of robust end-to-end architectures that can handle sequential dependencies without relying on error-prone intermediate transcription steps.

Availability: The dataset and codebase are publicly available on Hugging Face and GitHub, serving as a foundational resource for developing robust, culturally inclusive, and reliable speech agents.