💬 NLP

A Reliability Assessment of LALM Audio Judges for Full-Duplex Voice Agents

This paper empirically validates the reliability of Gemini 2.5 Flash as an audio judge for full-duplex voice agents by demonstrating its strong agreement with human raters across multiple dimensions, establishing a cost-effective basis for its deployment as a substitute or supplementary rater while cautioning that model performance varies within the Gemini family and requires specific re-validation.

Original authors: A. Sayyad, J. Emmons, S. Jones, T. Lin, H. Krishnan

Published 2026-07-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: A. Sayyad, J. Emmons, S. Jones, T. Lin, H. Krishnan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you run a busy voice-activated customer service line. Every time a customer talks to your AI agent, you want to know: Did the AI sound natural? Was the audio clear? Did it handle interruptions well?

Traditionally, to answer these questions, you'd hire a team of human "audio detectives." They listen to the recordings and give them a score from 1 to 5. But humans get tired, they get hungry, and they might disagree with each other. It's also incredibly expensive and slow.

So, the researchers at Salesforce asked a big question: Can we swap the human detectives for a super-smart AI judge (called a Large Audio Language Model, or LALM) to do the scoring?

They didn't just guess; they put the AI through a rigorous "driver's ed" test against a panel of three real human experts. Here is what they found, using the same 209 voice sessions (152 real conversations and 57 tricky, broken-up clips designed to trip up the AI).

The Main Discovery: The AI is a "Fourth Detective"

The study found that for 5 out of 8 specific things they were measuring (like how well the AI handled different accents or how natural its voice sounded), the AI judge agreed with the human panel almost as much as the humans agreed with each other.

Think of it like this: If you have three human judges, they might argue a little bit about a score. The AI judge stepped in, and on those 5 dimensions, its opinion was so close to the human average that it could easily be the fourth judge on the team.

The Proof: On 60% to 92% of the conversations, the AI's score was within just 1 point of the average human score. That's a pretty tight hug for a robot!
The Rank: The AI was also great at sorting conversations from "best" to "worst." If the humans said Conversation A was better than Conversation B, the AI usually said the same thing.

The "Gotchas": Where the AI Needs a Human Hand

However, the paper is very careful not to say "The AI is perfect." It explicitly rules out the idea that you can just swap the AI in and forget about humans forever. There are four specific areas where the AI needs a safety net:

The "Clarity" Blind Spot: When the audio was naturally clear, the AI and humans were fine. But when the audio was badly distorted (like a phone call with heavy static or a glitch), the AI sometimes missed the problem that the humans caught immediately. Specifically, if the audio was "clipped" (too loud and distorted) or had a weird sample rate, the AI often gave it a passing grade while the humans gave it a failing one.
- The Fix: The paper suggests using a simple, cheap computer program to check for these specific glitches first. If the program finds a glitch, send it to a human. If not, let the AI handle it.
The "Speed" Confusion: On two dimensions (how fast the AI spoke and the overall "fidelity" or faithfulness of the voice), the humans themselves couldn't agree on what the scores meant. Since the humans were confused, the AI was confused too.
- The Fix: Don't use the AI to score these two things yet. The problem isn't the AI; it's that the rulebook for humans needs to be rewritten first.
The "Model Swap" Trap: The researchers tested two newer AI models (Gemini 3.5 Flash and 3.1 Pro). One of them (3.5 Flash) was even better at agreeing with humans. But the other one (3.1 Pro) had a weird habit: it was good at ranking conversations (saying which was better), but it gave scores that were consistently 1 point lower than the humans.
- The Lesson: You can't just assume a new AI model will work the same way just because it's "smarter." You have to re-check its calibration (its scoring scale) before you let it loose.
The "Ceiling Effect": In many real conversations, the audio is so good that everyone gives it a perfect 5. When everyone gives a 5, it's hard to tell if the AI is actually "agreeing" or just guessing. The study showed that even when the AI agreed with humans 90% of the time, statistical tests sometimes said "no agreement" because there was no room to be wrong. The paper argues that looking at simple agreement (did they pick the same number?) is more useful than complex math in these cases.

The Bottom Line: A Massive Win for Speed and Cost

The paper concludes that for the dimensions where the evidence is strong, you can absolutely use the AI as your primary judge.

Why does this matter?

Cost: Hiring three humans to listen to 100 sessions takes about 35 hours of work (almost a full-time job). Using the AI costs just a few dollars in API fees. That's a cost reduction of roughly 100 times (two orders of magnitude).
Speed: With humans, you can only check a few sessions a week. With the AI, you could check 1,000 sessions a week without hiring a single new person.

The Verdict: The AI isn't a magic wand that replaces humans entirely, but it is a fantastic "fourth detective" that can do 90% of the heavy lifting. As long as you keep a human in the loop for the specific "glitchy" audio cases and fix the rulebook for the confusing dimensions, you can scale your voice quality checks up massively without breaking the bank. The paper suggests this is a defensible, real-world strategy, not just a cool experiment.

Technical Summary: A Reliability Assessment of LALM Audio Judges for Full-Duplex Voice Agents

Problem Statement

The paper addresses the need to validate Large Audio Language Models (LALMs) as automated judges for production voice agents. While previous literature has established correlations between LALMs and human ratings on isolated text-to-speech (TTS) utterances, there is a gap in evaluating LALMs on raw stereo audio from full-duplex, multi-turn agent conversations. The core question is whether LALM-generated ratings agree closely enough with human ratings to substitute for one or more human raters in a production evaluation pipeline. The authors frame substitutability not as a global property, but as an empirical, per-dimension characteristic.

Methodology

The study employed a rigorous validation design comparing a LALM against a three-rater human reference standard.

Dataset: A total of 209 stereo sessions were evaluated:
- 152 Natural Conversations: Full-duplex agent-client conversations spanning 13 accent-and-condition strata (covering American, Indian, British, French, and Italian accents, both clean and codec-degraded).
- 57 Adversarial Clips: Perturbed clips derived from 10 high-quality baselines, injected with six types of Digital Signal Processing (DSP) defects (e.g., amplitude clipping, dead air, Gaussian noise, truncation, overdubbing, sample-rate mismatch).
Dimensions: Ratings were generated across eight production dimensions: accent_dialect_handling, audio_clarity, speech_clarity, entity_pronunciation, prosody_naturalness, interruption_audio_quality, speaking_rate_adaptation, and overall_fidelity.
Judges:
- Human: Three calibrated annotators scored every session on a 1–5 Likert scale.
- LALM: Gemini 2.5 Flash (primary), Gemini 3.5 Flash, and Gemini 3.1 Pro Preview were tested. The models received raw stereo WAV bytes (agent left, client right) directly via the Vertex AI API, without transcription or intermediate feature extraction.
Metrics: The study utilized four agreement statistics:
1. Pairwise Spearman $\rho$ : For rank agreement (LALM vs. individual humans vs. human-human pairs).
2. Simple Agreement: The proportion of sessions where the LALM rating is within 1 point of the three-rater human mean.
3. Krippendorff $\alpha$ : For chance-corrected inter-rater reliability (noting its limitations in saturated distributions).
4. Recall Difference: Newcombe-Wilson 95% confidence intervals on the difference in defect detection sensitivity between LALM and humans in the adversarial arm.

Key Contributions and Results

1. Rank Agreement Parity

On 5 of 8 dimensions (accent_dialect_handling, entity_pronunciation, speech_clarity, prosody_naturalness, speaking_rate_adaptation), the LALM's rank correlation with humans ( $\rho$ ) departed from the average human-human correlation by at most 0.07. On 7 of 8 dimensions, the 95% bootstrap confidence intervals for LALM-human and human-human correlations overlapped.

Note: On audio_clarity, the LALM correlation (0.25) was roughly half the human-human correlation (0.47), though both were positive. On overall_fidelity, neither exhibited strong rank-discrimination signals.

2. Simple Agreement and the "Reliability Paradox"

Six of eight dimensions showed LALM agreement with the human mean within 1 point on 60% to 92% of sessions.

High Agreement: accent_dialect_handling (92.3%), audio_clarity (89.0%), and speech_clarity (90.9%).
Low Agreement: speaking_rate_adaptation (48.8%) and overall_fidelity (43.5%).
Statistical Insight: The authors highlight a "reliability paradox" where Krippendorff $\alpha$ was near zero or negative on many dimensions (e.g., -0.15 for speech_clarity) despite high simple agreement (>90%). This is attributed to "ceiling degeneracy," where ratings saturate near the top of the scale, reducing variance and rendering chance-corrected statistics uninformative. Consequently, the authors argue simple agreement is the appropriate lens for these production dimensions.

3. Adversarial Defect Detection

In the adversarial arm (48 defect-dimension cells):

The LALM was significantly more sensitive than humans in 4 cells.
Humans were significantly more sensitive in 3 cells.
The remaining 41 cells showed no significant difference (attributed to low statistical power due to small sample sizes, $n \le 8$ , rather than proven equivalence).
Blind Spots: The LALM failed to detect amplitude clipping and sample-rate mismatch on the audio_clarity dimension, whereas humans detected them consistently. Conversely, the LALM detected signal-to-noise degradation on accent_dialect_handling where humans missed it entirely.
Interpretation: The LALM and humans route the same acoustic evidence to different rubric constructs. For example, clipping was interpreted by the LALM as a prosody or fidelity issue, while humans categorized it as an audio clarity issue.

4. Cross-Model Replication

The study tested Gemini 3.5 Flash and Gemini 3.1 Pro Preview:

Gemini 3.5 Flash: Improved simple agreement to 8 of 8 dimensions, maintaining rank correlation parity on 4 of 8 dimensions.
Gemini 3.1 Pro Preview: Maintained rank correlation parity but exhibited a calibration offset, rating audio_clarity and speaking_rate_adaptation more than 1.5 points below the human mean. This demonstrates that rank correlation alone is insufficient to validate a model swap; simple agreement (calibration) must be re-verified.

5. Architectural Consistency

The LALM demonstrated superior intra-judge consistency compared to human raters, who showed substantial between-rater variation (differences of >1 point on some dimensions). The LALM tends toward the central tendency of the human panel, offering a stable, reproducible headline metric.

Significance and Claims

The paper claims to provide the first empirical evidence of a production-grade, audio-native LALM judge being validated against a multi-rater human standard on full-duplex conversational audio.

Deployment Viability: The evidence supports deploying the LALM (specifically Gemini 2.5 Flash) as a substitute or fourth rater on 5 of 8 dimensions where rank agreement is high, and on 6 of 8 dimensions where simple agreement is high.
Cost Efficiency: The authors estimate that replacing human rating with LALM reduces operational costs by two orders of magnitude (from ~0.88 FTE to a few dollars per week in API costs) and unlocks throughput, allowing for weekly audits of 1,000+ sessions rather than being capped by rater hours.
Caveats and Safeguards: The authors explicitly identify four areas requiring caution before full deployment:
1. Audio Clarity: Requires human spot-checks for sessions rated below 4 due to lower rank-discrimination in degraded audio.
2. Specific Artifacts: A lightweight DSP detector is recommended to route amplitude clipping and sample-rate mismatch cases to human review.
3. Rubric Validity: Overall_fidelity and speaking_rate_adaptation should be dropped from headline metrics until rubric definitions are refined, as human raters themselves disagree on these dimensions.
4. Model Swaps: Future model updates must undergo a specific "simple-agreement" calibration check, as rank correlation does not guarantee score alignment.

The paper concludes that while the LALM is not a perfect replacement in all scenarios, it provides a defensible empirical basis for a phased deployment strategy that significantly reduces cost while maintaining quality control through targeted human oversight.