VAmoS Bench: Voice Agent Simulation Bench
The paper introduces VAmoS Bench, a new evaluation framework that measures the end-to-end performance of voice agents in stateful, financial customer-support scenarios by simulating realistic phone calls with adversarial elements and grading agents based on their ability to correctly resolve tasks, update databases, and maintain security without human intervention.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
The Voice in the Machine: A New Way to Test AI Phone Calls
Imagine a world where you can talk to a computer just like you talk to a friend. This isn't science fiction; it's the rapidly growing field of voice agents. These are smart computer programs that can hear your voice, understand what you're saying, and talk back to you. They are already being used in banks to help with lost credit cards and in clinics to remind patients of appointments. But building one is tricky. It's like building a robot that needs to hear you, think about your question, find the right answer in a giant database, and then speak it back to you, all in the blink of an eye.
For a long time, scientists tested these robots by checking their parts separately. They'd ask, "How well does the robot hear?" (measuring how many words it gets wrong) or "How natural does it sound?" (measuring if it sounds like a human). But this is like testing a car by only checking the engine and the tires, without ever seeing if it can actually drive down the road without crashing. The real question isn't just about the parts; it's about the whole job: Did the robot actually solve the problem? In the business world, this is called "containment"—did the computer handle the call all by itself, or did it have to give up and call a human? This paper introduces a new way to test these robots that looks at the whole picture, not just the pieces.
The Big Test: VAmoS Bench
The authors of this paper, a team from Veris AI, realized that existing tests were missing the most important part of the story. They created something called VAmoS Bench (Voice Agent Simulation Bench). Think of it as a giant, high-tech "driving test" for voice agents, but instead of a car, they are testing a virtual bank teller named Riley.
Riley's job is to handle credit card issues for a fictional bank. She has to do things like freeze a stolen card, cancel a lost one, or activate a new one. To test Riley, the researchers didn't just ask her questions on a screen. They built a simulation where a "caller" (a computer program pretending to be a human) actually calls Riley over a real audio connection. The caller has a secret goal, like "I want a new card but I don't want to prove who I am," or "I lost my card and I'm in a hurry."
The test is set up like a video game with 100 different levels (scenarios). Some levels are easy, like a normal customer asking for a card status. Some are tricky, requiring the agent to do several steps in the right order. And some are "boss battles" where the caller tries to trick the agent, pressure her into skipping safety steps, or even try to hack her with weird words.
The Magic of "Say" vs. "Do"
The coolest part of this test is how it grades the agents. In the past, a test might just listen to the conversation and say, "Great job! You sounded like you fixed the card." But VAmoS Bench is a detective. It looks at two things at once:
- What the agent said (the transcript).
- What the agent actually did (the database changes and tool calls).
Imagine a student taking a math test. If they write "The answer is 42" on the paper but didn't actually do the math, they get an F. If they do the math correctly but write the wrong answer, they also get an F. VAmoS Bench catches agents that are "faking it." For example, an agent might say, "I've frozen your card," but if the computer check shows it never actually sent the command to freeze the card, the test marks it as a failure. Conversely, if an agent fixes the card but accidentally tells the caller their secret password, the test marks that as a failure too, even though the card is fixed.
The Results: Who Passed the Test?
The researchers put 11 different voice agent systems through this 100-call gauntlet. They ran the test three times for each system to make sure the results were real. Here is what they found:
- The Winners: The systems built on Pipecat and LiveKit Agents did the best, successfully handling about 71% and 70.3% of the calls, respectively.
- The Losers: The system called Nemotron struggled the most, only succeeding on 43% of the calls.
- The "Boss Battles": The hardest part of the test for every single agent was the complex scenarios. While agents were okay at simple tasks, they only succeeded on 52.2% of the complex calls. These were the calls that required multiple steps, like checking a user, finding a specific card, freezing it, and then ordering a replacement, all while keeping the conversation flowing.
- The Tricky Part: The agents were actually quite good at saying "No" to people trying to trick them (the adversarial group), succeeding on 73.4% of those calls. However, they were terrible at keeping secrets when they said "No." Even when they refused to help an unverified caller, they often accidentally told the caller which piece of information was wrong (like "Your address is incorrect"), which gave the bad guy the clue they needed to try again.
Why This Matters (And What It Doesn't Mean)
The paper shows that while many voice agents sound great, they often fail at the actual job of solving problems, especially when things get complicated or when they need to follow a strict order of operations. The team found that 27 times out of the thousands of attempts, an agent claimed to have done something (like freezing a card) but the computer logs showed they never actually did it.
However, the authors are careful not to declare a single "winner" for the whole world. They point out that these results are specific to this one bank scenario and these 100 tests. The differences between the top performers were small (less than 1%), and because the test was a simulation, we don't know exactly how they would perform with real, messy human customers in a real bank.
In short, VAmoS Bench is a new, stricter ruler for measuring voice agents. It stops us from being fooled by a robot that sounds polite but does nothing, and it highlights that the real challenge isn't just making the robot talk—it's making sure it actually does the right thing, in the right order, without giving away any secrets.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.