💬 NLP

Do Audio Language Models Use Paralinguistic Evidence? Counterfactual Audits for Response Evaluation

This paper introduces counterfactual audits to evaluate whether audio-language models genuinely utilize paralinguistic cues for response assessment, revealing that standard accuracy metrics often mask distinct failure modes and overstate reliability across various models.

Original authors: Kevin Miller, Arjun Chandra, Venkatesh Saligrama

Published 2026-08-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Kevin Miller, Arjun Chandra, Venkatesh Saligrama

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to be a helpful voice assistant. You want it to not just hear what you say, but also how you say it. If you ask for directions while sounding frantic and angry, a good assistant should calm you down. If you ask the same question while sounding happy and relaxed, the assistant should be cheerful. This ability to read the "vibe" of a voice—the tone, the speed, the pitch—is called paralinguistics. It's the difference between a robot that hears words and one that truly listens.

Recently, scientists have started using super-smart computer brains called Audio-Language Models (ALMs) to act as judges. Instead of hiring humans to listen to thousands of conversations and grade them, they let these AI judges decide which assistant response was better. But here's the catch: just because a computer can hear audio doesn't mean it actually understands the emotion in the voice. It might be relying on the text transcript or guessing based on the words used, ignoring the tone entirely. This paper asks a critical question: Are these AI judges actually listening, or are they just pretending?

The Great Voice Detective Audit

The researchers from Boston University decided to put these AI judges through a "counterfactual audit." Think of this like a magic trick where the script stays exactly the same, but the actor's performance changes.

Imagine you have a script that says, "I'm fine."

Scenario A: You say it with a cheerful, bouncy voice. The perfect response is, "Great! What's next?"
Scenario B: You say the exact same words with a flat, sad, or angry voice. The perfect response is, "Are you sure? You sound upset."

The researchers created thousands of these "magic trick" pairs. They kept the words identical but changed the audio recording to have different emotions or different timing for when the emotion shifted. Then, they asked the AI judges to pick the right response for the audio they heard.

The "Two-Choice" Shortcut

The most surprising discovery was that many of the top AI judges are like students who can solve a math problem if they see the answer choices side-by-side, but fail miserably if they have to solve it alone.

The researchers tested the judges in two ways:

The "Native" Test (One Context): The judge hears one audio clip and has to pick the best response from two options. This is how the judge would work in the real world.
The "Contrastive" Test (Two Contexts): The judge hears both audio clips (the happy one and the sad one) and both responses at the same time, and has to match them up.

The results were eye-opening. When the judges were allowed to see both options at once (the Contrastive Test), many of them, like the Gemini family of models, got really high scores (sometimes over 90%). They could clearly tell the difference between the happy and sad voices when the contrast was right in front of them.

However, when the researchers switched to the "Native" test—where the judge had to make a decision based on just one audio clip, just like a real user would experience—the performance of these same models crashed. They dropped down to near-random guessing levels (around 50-55%).

This suggests that these models have a "shortcut" ability: they can spot the difference when forced to compare, but they fail to use that skill when they have to rely on it alone. It's like a student who can spot the correct answer on a multiple-choice test by eliminating the wrong ones, but if you ask them to write the answer from memory, they draw a blank.

The "Potemkin" Illusion

The authors call this a "Potemkin failure." Imagine a village that looks beautiful from the outside but is hollow inside. These AI judges look smart because they can pass the "two-choice" test, but inside, they aren't actually using the audio cues to make their decisions in real-time.

To figure out exactly why they were failing, the researchers broke the task down into three steps, like a relay race:

Perception: Did the AI hear the emotion? (e.g., "Is the user angry?")
Mapping: If the AI knew the user was angry, could it pick the right text response?
Judgment: Could the AI put it all together in the real-world scenario?

They found that for some models, the problem wasn't hearing the emotion or knowing the right text. The problem was orchestration. They could do the individual steps if asked directly, but they couldn't combine them to make a final decision in the native setting. It's like having a chef who can chop vegetables perfectly and season a steak perfectly, but when asked to cook the whole meal, they forget to turn on the stove.

The Timing Trap

The paper also tested a harder version of the game: Positional Emotion. Imagine a conversation where the user starts happy but gets frustrated halfway through because the assistant made a mistake. The correct response depends on when the user got mad.

In this complex scenario, the AI judges struggled even more. Even the best models, which could handle simple "happy vs. sad" single-turn tests, often failed to track the timing of the emotion shift in a long conversation. They seemed to get lost in the timeline, unable to pinpoint exactly when the mood changed and why.

The Bottom Line

The paper concludes that we cannot trust these AI judges just because they get high scores on standard tests. High accuracy numbers can hide the fact that the model is failing in specific, dangerous ways. Some models are "Potemkin" judges—impressive on the surface but hollow in practice. Others are "shortcut" judges that rely on text clues rather than listening to the voice.

The researchers suggest that before we let these AI models judge voice assistants in the real world, we need to run these specific "counterfactual audits" to make sure they are actually listening to the tone of voice, and not just reading the script. Until then, we might be trusting a robot that thinks it's listening, but is really just guessing.

Technical Summary: Counterfactual Audits for Paralinguistic Response Evaluation in Audio-Language Models

1. Problem Statement

Audio-Language Models (ALMs) are increasingly deployed as automatic judges for speech-to-speech systems, tasked with evaluating whether an assistant's response is appropriate given a user's input. While text-based evaluation is well-established, spoken interaction relies heavily on paralinguistic cues (tone, pitch, rhythm, timing) that often disambiguate the user's intent. A user may say the same words with frustration, hesitation, or surprise, requiring distinct responses.

The core problem addressed is the second-order evaluation gap: before an ALM can be trusted as a judge for paralinguistic appropriateness, we must verify that the judge itself actually utilizes audio evidence rather than relying on lexical shortcuts, transcript content, or generic conversational preferences. Current benchmarks often fail to diagnose how a judge fails, conflating perception errors with reasoning errors.

2. Methodology: Counterfactual Audits and Diagnostic Cascade

The authors introduce a framework of counterfactual audits designed to isolate paralinguistic reasoning. The methodology holds the lexical content (transcript) constant while varying the paralinguistic realization (affect, prosody, or timing of affective shifts).

2.1 Audit Item Construction

Each audit item is a tuple $E = (C, A_0, A_1, R_0, R_1)$ :

$C$ : A shared transcript or conversation history.
$A_0, A_1$ : Two audio realizations of the same words with different paralinguistic states (e.g., angry vs. sad).
$R_0, R_1$ : Candidate responses appropriate for $A_0$ and $A_1$ respectively.

The study utilizes two task families:

SINGLE-TURN-EMOTIONS: Static calibration isolating paralinguistic response selection without dialogue history or causal attribution.
POSITIONAL-EMOTION: Multi-turn tasks where the transcript is fixed, but the timing of an affective shift (e.g., when a user becomes frustrated) varies. This tests temporal-causal reasoning, as the correct response depends on identifying the specific turn that triggered the emotional shift.

2.2 The Diagnostic Cascade

To move beyond aggregate accuracy, the authors decompose the evaluation into a root-cause diagnostic cascade (Figure 1), evaluating three binary states for each item:

$P$ (Perception): Does the judge correctly identify the paralinguistic state (e.g., "The user is angry")?
$O$ (Oracle Response Mapping): If the paralinguistic state is provided explicitly in text (removing the audio perception burden), can the judge map that state to the correct response?
$J$ (Native Judgment): Does the judge succeed in the standard deployment setting (Pointwise: one audio context, two responses)?

These states form a diagnostic tuple $(P, O, J)$ , allowing the authors to categorize failures into distinct modes (Table 1):

Potemkin Failure ($1, 1, 0$): The judge perceives the cue and knows the mapping in isolation but fails to compose these skills in the native audio judgment.
Perception Bottleneck ($0, 1, 0$): The judge fails to hear the cue but can map the state if told.
Response-Mapping Bottleneck ($1, 0, 0$): The judge hears the cue but cannot map it to the right response.
Reliable Integrated Judgment ($1, 1, 1$): All components succeed.

2.3 Evaluation Protocols

Pointwise (Native): The judge receives one audio context and two candidate responses. This mimics the deployment setting.
Pairwise (Contrastive Recoverability): The judge receives both audio contexts and both responses and must match them. This tests if the model possesses the latent ability to distinguish the counterfactuals when the contrast is explicit.

3. Key Results

The authors evaluated a range of proprietary (Gemini, GPT, Nova) and open-source (Qwen, Phi, Voxtral, DeSTA) ALMs.

3.1 Contrastive Success vs. Native Reliability

A primary finding is that contrastive recoverability often overstates native judge reliability.

Many models (particularly Gemini variants) achieve high accuracy in the Pairwise setting (e.g., Gemini-3-Pro reaches 91.0% on Single-Turn Pairwise with Hard Cues).
However, these same models degrade significantly in the Pointwise setting (e.g., dropping to 65.3% for the same model).
This "protocol collapse" indicates that while models can distinguish differences when both options are visible, they fail to deploy this ability when forced to make a decision based on a single audio context.

3.2 Heterogeneous Failure Modes

Aggregate accuracy masks distinct failure mechanisms:

Gemini Models: Exhibit high "Potemkin" failure rates. They often pass perception and oracle mapping probes but fail the final integration step in the native audio setting. For instance, Gemini-3-Pro has a 26% Potemkin mass in single-turn tasks.
GPT Models: Show lower reliable integration and larger perception bottlenecks, failing earlier in the pipeline before the mapping question becomes relevant.
Positional Tasks: Performance drops across all models in multi-turn temporal tasks. Even the strongest models struggle with the "Positional Pointwise" setting (often near chance, ~50%), indicating that temporal-causal paralinguistic evaluation is a significant challenge.

3.3 Validity Checks

The authors conducted robustness checks to ensure results were not artifacts of synthetic speech or generator bias:

Synthetic vs. Real Speech: Replacing synthesized audio with real human speech (from CAVA) yielded highly correlated diagnostic profiles ( $\rho > 0.9$ ).
Generator Bias: Substituting the model used to generate the dataset annotations did not alter model rankings.
Quality Balance: Irrelevant factors like audio quality (DNSMOS) and lexical helpfulness were balanced between candidate responses, ruling out confounding variables.

4. Contributions

Counterfactual Audits: Introduction of a framework using counterfactual audio items to explicitly test whether ALM judges rely on paralinguistic evidence rather than lexical shortcuts.
Root-Cause Diagnostic Procedure: A novel decomposition of judge failure into Perception ( $P$ ), Oracle Mapping ( $O$ ), and Native Judgment ( $J$ ) states, identifying specific bottlenecks like "Potemkin" failures.
Empirical Audit of Frontier Models: A comprehensive evaluation showing that aggregate accuracy is insufficient for certifying audio-judge reliability, particularly highlighting the fragility of temporal-causal reasoning in current models.

5. Significance and Claims

The paper argues that ALM judges should not be evaluated by accuracy alone. The existence of a large gap between Pairwise (contrastive) and Pointwise (native) performance suggests that models may possess latent paralinguistic abilities that they fail to activate in standard deployment scenarios.

The authors conclude that:

Contrastive success does not imply native judge reliability. A model that can match two options when both are present may still fail to judge a single interaction correctly.
Aggregate accuracy hides mechanism. Two models with similar accuracy scores may have fundamentally different failure modes (e.g., one failing at perception, another at orchestration).
Thorough behavioral audits are required. Before deploying ALMs as judges for speech-to-speech systems, they must undergo these diagnostic audits to ensure they are not merely "Potemkin" evaluators that appear to understand audio cues but rely on text-based heuristics.

The work leaves the improvement of judge performance (via training or data design) to future work, focusing instead on the rigorous identification of failure modes.