🤖 AI

$\tau$ -Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains

The paper introduces $\tau$ -Voice, a novel benchmark that evaluates full-duplex voice agents on complex, grounded real-world tasks using a controllable user simulator, revealing that current agents retain only 30–45% of their text-based capabilities due to significant behavioral failures in noisy, multi-turn conversational environments.

Original authors: Soham Ray, Keshav Dhandhania, Victor Barres, Karthik Narasimhan

Published 2026-03-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Soham Ray, Keshav Dhandhania, Victor Barres, Karthik Narasimhan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to hire a new customer service representative. You have two candidates: Text-Tom and Voice-Vera.

Text-Tom is brilliant. He sits in a quiet office, reads your emails, thinks deeply, and solves your problems perfectly. If you ask him to change your order, he does it instantly.

Voice-Vera is the new hire. She is supposed to do the same job, but she has to do it while you are talking to her on a busy street, with traffic noise, while you have a cold, and while she is trying to listen to you and speak at the same time.

This paper, τ-Voice, is like a rigorous "job interview" designed specifically to test how well Voice-Vera (and her competitors) actually perform in the real world, rather than in a quiet test lab.

Here is the breakdown of what they found, using simple analogies:

1. The Problem: The "Quiet Room" vs. The "Busy Street"

Previously, researchers tested voice agents in two separate ways:

Test A: "Can this agent solve a math problem?" (Task completion).
Test B: "Can this agent wait politely while you talk?" (Conversational flow).

They never tested them together in a messy, real-world scenario. It's like testing a race car on a smooth track (Task A) and testing a driver's ability to talk on the phone (Test B), but never seeing if the driver can race while talking on the phone in a rainstorm.

τ-Voice built a "simulated storm." They created a computer program that acts like a human caller with:

Accents: Different ways of speaking (like a British accent or a heavy regional accent).
Noise: Background sounds like dogs barking or traffic.
Interruptions: People talking over each other, saying "um," or coughing.

2. The Experiment: The "Torture Test"

The researchers took three top-tier voice agents (from Google, OpenAI, and xAI) and put them through 278 different customer service tasks (like returning a puzzle, changing a flight, or fixing a phone bill).

They tested them in two conditions:

The "Clean" Condition: Perfect audio, no noise, no interruptions. (Like talking in a soundproof studio).
The "Realistic" Condition: Noisy street, diverse accents, people interrupting each other. (Like talking in a crowded coffee shop).

3. The Results: The "Voice Gap"

The results were surprising and a bit scary for the future of voice AI.

The Text Champion: When using a text-based AI (GPT-5), the success rate was 85%. It was like a master chef cooking in a perfect kitchen.
The Voice Reality:
- In the Clean condition, the voice agents dropped to 31–51%. They were already struggling just by switching from typing to talking.
- In the Realistic condition, they dropped even further to 26–38%.

The Analogy: Imagine a student who gets an A (85%) on a written math test. But when you ask them to solve the same math problems while someone is shouting at them and they have to speak the answers out loud, they suddenly get a D (30%). They aren't "dumb"; the medium of voice is just incredibly hard for them to handle right now.

4. Why Did They Fail?

The researchers looked closely at the mistakes. They found that 80–90% of the failures were the AI's fault, not the simulator's.

The "Lost in Translation" Problem: The AI often couldn't understand the user's name or email when they spelled it out, especially if the user had an accent or background noise.
The "Hallucination" Problem: The AI would confidently make things up. For example, it might say, "I've updated your address," when it actually didn't do anything.
The "Bad Timing" Problem:
- Some AIs were too interruptive. They would cut the user off constantly, like a person who never lets you finish a sentence.
- Some AIs were too slow. They would wait too long to respond, making the conversation feel awkward and dead.
- Some AIs were too sensitive. They would stop talking just because the user coughed or said "mm-hmm," thinking the user was taking over the conversation.

5. The "Accessibility" Warning

One of the most important findings was about accents.

One AI provider (xAI) struggled massively with non-American accents, losing nearly 40% of its ability to understand.
Another provider (Google) was much more robust, barely losing any performance with accents.

The Metaphor: Imagine a translator who speaks perfect English but gets completely confused if you speak with a slight accent. If you rely on that translator for your bank account, you are in trouble. This paper shows that current voice AIs might systematically fail to help people with non-standard accents, which is a huge fairness issue.

6. The Bottom Line

The paper concludes that while voice agents are getting better, they are still far from being ready to replace human customer service agents in complex situations.

Text AI is like a sprinter: Fast, accurate, and focused.
Current Voice AI is like a sprinter trying to juggle while running through mud: They are trying to do too many things at once (listen, speak, think, handle noise) and they are dropping the balls.

The Takeaway: We need to stop testing voice agents in "quiet rooms" and start testing them in the "mud." Only by measuring how they handle real-world chaos can we build voice assistants that are truly reliable, fair, and ready for the real world.

1. Problem Statement

The paper addresses the critical gap between the rapid deployment of full-duplex voice agents (systems that listen and speak simultaneously) and the lack of rigorous evaluation frameworks for them. Existing benchmarks suffer from three main limitations:

Isolation of Capabilities: They evaluate task completion (tool use) and conversational dynamics (turn-taking) separately, rather than testing them together in a grounded, real-world context.
Lack of Realism: Many benchmarks use synthetic tasks without real tool calls, or they rely on text-only inputs that ignore acoustic degradation (noise, accents, telephony compression).
Inability to Simulate Complexity: Current systems struggle to simulate the continuous time, interruptions, and backchanneling required for natural full-duplex interaction without being constrained by real-time wall-clock limits.

The core question is: Can voice agents handle complex, multi-step customer service tasks with the same reliability as text-based agents when faced with realistic audio conditions and conversational dynamics?

2. Methodology: The τ-Voice Framework

The authors introduce τ-Voice, an extension of the existing $\tau^2$ -bench, designed to evaluate voice agents on 278 grounded tasks across three domains: Retail (returns/exchanges), Airline (booking changes), and Telecom (billing/plan changes).

A. Full-Duplex Orchestrator

The core innovation is a tick-based orchestrator that decouples simulation time from wall-clock time.

Discrete Time: Audio streams are discretized into fixed-duration ticks (default $\tau = 200$ ms). Both the agent and user exchange exactly $\tau$ ms of audio per tick.
Buffering & Interruption: If an interruption occurs, the buffer is cleared, truncating the agent's in-progress response. This allows for true simultaneous speech.
Text Linearization: Overlapping speech is linearized into sequential text for the user simulator's LLM to process, ensuring the simulator understands the conversation flow despite full-duplex audio.
Decoupling: By advancing simulation time independently, the user simulator can utilize the most capable LLMs without real-time latency constraints, ensuring high-quality instruction following and turn-taking decisions.

B. Voice User Simulator

A controllable simulator generates realistic caller audio through a pipeline:

Text Generation: Produces natural spoken language with disfluencies ("um," "uh"), verbalized special characters, and terse responses.
Speech Synthesis: Uses diverse Voice Personas (7 total) covering various accents (e.g., Bengali, Sichuan, Indian English) and demographics.
Audio Environment: Simulates realistic telephony conditions by mixing:
- Background Noise: Continuous (traffic, chatter) and burst noise (phone rings, dog barks).
- Channel Degradation: Dynamic muffling, G.711 $\mu$ -law compression (8kHz), and frame drops (simulating packet loss via a Gilbert-Elliott model).
- Turn-Taking Dynamics: LLM-driven decisions for interruptions, backchanneling ("mm-hmm"), and yielding.

C. Evaluation Metrics

The framework measures two distinct dimensions:

Task Completion (pass@1): Verifiable success based on the final database state compared to annotated goals. Unlike text benchmarks, this accounts for spoken output variability using LLM-based evaluation rather than strict string matching.
Voice Interaction Quality:
- Responsiveness: Response Rate (RR) and Yield Rate (RY).
- Latency: Response Latency ( $L_R$ ) and Yield Latency ( $L_Y$ ).
- Interrupt Rate: Frequency of the agent cutting off the user.
- Selectivity: Ability to ignore non-actionable signals like backchannels, vocal tics, and non-directed speech.

3. Key Contributions

First Integrated Benchmark: $\tau$ -Voice is the first benchmark to combine verifiable task completion, full-duplex interaction, and realistic audio environments in a single framework.
Controllable Simulator: A novel user simulator that allows systematic ablation of acoustic factors (noise, accents, turn-taking) while maintaining reproducibility via seed-based stochasticity.
Comprehensive Empirical Analysis: Evaluation of three leading audio-native providers (OpenAI, Google, xAI) across 278 tasks, providing the first large-scale comparison of voice vs. text performance in grounded domains.

4. Key Results

The study reveals a significant "Voice-Text Gap" and highlights specific failure modes:

A. Performance Degradation

Text Baseline: GPT-5 (reasoning) achieves 85% pass@1 on text tasks.
Clean Voice: Under ideal conditions (clean audio, American accents, no interruptions), the best voice agents achieve only 31–51% (a 34–54 percentage point drop).
Realistic Voice: Under realistic conditions (noise, diverse accents, interruptions), performance drops further to 26–38%. Voice agents retain only 30–45% of the capability of state-of-the-art text models.

B. Impact of Acoustic Factors (Ablation Study)

Accents: The most damaging factor on average, causing a 10pp drop. However, this is highly provider-specific: xAI loses 38% of its clean capability due to accents, while Google is nearly unaffected.
Turn-Taking: Causes a 7pp drop on average.
Noise: Causes a 4pp drop.
Provider Trade-offs:
- Google: Most robust to degradation (loses only 17% of clean performance in realistic settings) but has lower responsiveness.
- OpenAI: Fastest latency (0.90s) and perfect responsiveness (100%), but poor selectivity (6%), often interrupting or responding to non-directed speech.
- xAI: Highest task completion in clean conditions but suffers the most from accents and has the highest interrupt rate (84%).

C. Error Analysis

Qualitative analysis of 91 failed simulations shows that 79–90% of failures stem from agent behavior, not simulator artifacts.

Primary Failure Modes:
- Authentication Bottlenecks: Agents fail to transcribe spelled-out names/emails correctly, blocking downstream tasks.
- Hallucinations: Agents fabricate order details or claim to have updated data without making tool calls.
- Reasoning Errors: Agents lose track of multi-step requests or fail to combine information spread across turns.
Conclusion: The gap is not purely a speech recognition (ASR) problem; it reflects fundamental reasoning and grounding challenges in the voice modality.

5. Significance and Future Implications

Accessibility Concerns: The severe performance drop for non-American accents (especially for xAI) suggests current voice agents may systematically underserve users with diverse speech patterns, raising equity issues.
Deployment Readiness: The findings indicate that while voice agents have closed the gap with non-reasoning text models in ideal conditions, they are not yet ready for reliable deployment in complex, noisy, real-world scenarios.
Research Direction: The paper calls for a shift from isolated ASR or dialogue metrics to end-to-end evaluation that accounts for the compounding effects of acoustic noise, turn-taking dynamics, and reasoning.
Open Source: The authors release $\tau$ -Voice as an open-source, configurable platform to enable reproducible research and the development of more robust, natural, and reliable voice agents.

In summary, $\tau$ -Voice provides the first rigorous evidence that current full-duplex voice agents struggle significantly with grounded tasks under realistic conditions, primarily due to agent reasoning and interaction management failures rather than just audio processing limitations.