Technical Summary: Real-Time Voice AI Hears but Does Not Listen

Problem Statement

Current real-time voice AI systems operate under a critical limitation: while they process speech as an audio stream, their decision-making logic effectively reduces the input to a text transcript, ignoring the non-lexical channel (pitch, tone, emotion, accent, and age). In human communication, the lexical channel (words) and the non-lexical channel (delivery) often carry complementary or conflicting information. In high-stakes scenarios—such as emergency dispatch, financial fraud detection, or volunteer recruitment—a decision based solely on words can be dangerous if the delivery signals distress, coercion, or insincerity. The paper identifies a disconnect between the system's ability to perceive these vocal cues and its ability to act upon them, terming this the "emotional intelligence gap of voice AI."

Methodology

The authors evaluated four leading production real-time voice systems:

1. OpenAI: GPT Realtime 2
2. Google: Gemini 3.1 Flash Live
3. Alibaba: Qwen3.5 Omni Plus Realtime
4. Alibaba: Qwen3.5 Omni Flash Realtime

The study utilized synthesized speech generated by ElevenLabs to ensure controlled variables. The evaluation consisted of two distinct experimental phases:

1. Multi-Turn Scenario Calls (Action-Oriented)

Three consequential scenarios were designed where the caller's words and delivery pointed to opposite actions. The system acted as the agent (dispatcher, bank officer, recruiter).

• Welfare Callback: A caller insists "nothing is wrong" while crying. Expected Action: Initiate safety response. Scripted Action: Close call.
• Wire-Fraud Check: A caller authorizes a transfer in a frightened voice. Expected Action: Escalate due to potential duress. Scripted Action: Approve transfer.
• Volunteer Recruitment: A caller agrees to sign up in a sarcastic tone. Expected Action: Do not enroll. Scripted Action: Enroll.

The systems were tested under base prompts, as well as with "attend" instructions (pay attention to delivery) and "override" instructions (do not act on words alone if delivery signals distress/coercion).

2. Single-Turn Diagnostics (Perception-Oriented)

To separate perception from action, the authors tested the systems in isolation:

• Delivery Detection: Asking if a speaker sounds distressed, frightened, or sarcastic given the same words spoken in conflicting tones.
• Speaker Property Identification: Asking the model to identify a speaker's accent (when the script discusses a different country) and age (when an adult voice reads child-like lines).

Human listeners validated the synthesized stimuli to ensure the intended non-lexical cues were audible.

Key Results

1. Actions Follow Words, Not Delivery

Across all three consequential scenarios, all four systems predominantly acted on the lexical content, disregarding the vocal delivery:

• Welfare Callback: All systems ended the call for crying callers who insisted everything was fine (5/5 runs for all models).
• Wire-Fraud Check: Systems approved transfers authorized in frightened voices at rates nearly identical to calm voices (4-5/5 runs).
• Volunteer Recruitment: Systems enrolled callers speaking sarcastically in 5/5 runs, failing to detect the insincerity.

2. Perception Exists but is Ignored (The Gap)

A critical finding is that for three of the four systems (GPT Realtime 2, Gemini Live, Qwen3.5 Omni Plus), the failure is not a lack of perception but a lack of integration into decision-making:

• Diagnostic Accuracy: When asked directly, these three systems reliably identified distress, fear, and sarcasm in the audio (e.g., 20/20 runs for distress detection) significantly outperforming text-only baselines.
• The Disconnect: Despite correctly identifying the emotion in diagnostics, these same systems ignored it during the multi-turn decision-making tasks.
• Exception: Qwen3.5 Omni Flash struggled to perceive the cues even in diagnostics (e.g., misidentifying sincere voices as sarcastic) and consequently failed to act correctly.

3. Accent and Age Biases

The systems exhibited a strong bias toward the semantic content of the script rather than the acoustic properties of the voice:

• Accent: When an Australian voice read a script about Italy, most systems (including GPT Realtime 2 and Qwen3.5 Omni Flash) identified the accent as "Italian," citing acoustic features that were not present. Qwen3.5 Omni Plus was the only model to partially recover the true accent for several speakers.
• Age: When adult voices read lines written for children, most systems estimated the speaker's age as 4–6 years old, driven by the script. Gemini Live was the only model to occasionally estimate the adult age correctly.

4. Limited Efficacy of Prompting

Explicit instructions to "attend" to delivery or "override" word-based decisions yielded inconsistent and partial results:

• Wire-Fraud: The "override" instruction improved escalation rates for frightened voices in some models (e.g., Gemini Live escalated 5/5 times), but systems still approved calm transfers.
• Welfare & Recruitment: Instructions largely failed to change the outcome; systems continued to close calls for crying callers or enroll sarcastic volunteers despite the prompts.

Key Contributions

1. Empirical Evidence of the Emotional Intelligence Gap: The paper provides the first systematic evaluation showing that leading production real-time voice systems act as if speech has been reduced to a transcript, even when they possess the capability to perceive vocal nuances.
2. Decoupling Perception and Action: By separating diagnostic queries from consequential decision tasks, the study demonstrates that the failure is often architectural or behavioral (ignoring the cue) rather than purely perceptual.
3. Evaluation Framework: The authors propose a methodology that deliberately creates conflicts between lexical and non-lexical channels to test safety, arguing that transcript-based evaluation is insufficient for voice AI.

Significance and Claims

The authors claim that current real-time voice AI systems are unsafe for deployment in settings where tone and emotion carry decisive information (e.g., healthcare triage, financial security, emergency services). The "emotional intelligence gap" represents a fundamental limitation where the system's internal representation of the speaker's state does not influence its external actions.

The paper concludes that while prompting can surface some caution, it does not reliably close this gap. Consequently, these systems should be deployed with extreme caution in regulated or high-stakes environments until the discrepancy between what the AI "hears" and what it "listens to" is resolved. The findings suggest that without addressing this gap, voice AI risks automating harmful decisions by prioritizing literal text over human context.