💬 NLP

Still Between Us? Evaluating and Improving Voice Assistant Robustness to Third-Party Interruptions

This paper addresses the vulnerability of Spoken Language Models to Third-Party Interruptions by introducing the TPI-Train dataset and TPI-Bench evaluation framework, which collectively mitigate semantic shortcut learning and enforce acoustic cue prioritization to enable robust multi-party spoken interactions.

Original authors: Dongwook Lee, Eunwoo Song, Che Hyun Lee, Heeseung Kim, Sungroh Yoon

Published 2026-04-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dongwook Lee, Eunwoo Song, Che Hyun Lee, Heeseung Kim, Sungroh Yoon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are talking to a very smart, helpful robot butler in your living room. You ask, "What's the weather like?" Just as you finish, your friend walking by chimes in, "Don't forget to check if the garage door is closed!"

In a perfect world, the robot would understand that you asked about the weather, but your friend is talking about the garage. It would answer your question and maybe add a note about the garage.

But in the real world, today's voice assistants (like Siri, Alexa, or advanced AI models) often get confused. They hear two voices, but their brains treat it like one person talking to them. They might think you said, "What's the weather like? Don't forget to check if the garage door is closed!" and give you a weird, nonsensical answer that mixes the two topics together.

This paper is about teaching these robots how to stop getting confused when a third person interrupts.

The Problem: The "One-Brain" Robot

Currently, voice assistants are great at talking to one person at a time. But if someone else jumps into the conversation, the robot loses its mind. It doesn't listen to who is speaking; it just listens to what is being said.

Think of it like a student taking a test who only reads the words on the page but ignores the fact that two different people are whispering answers to them. The student (the AI) just combines the whispers into one long sentence and writes it down, even though it makes no sense.

The Solution: A New Training Camp (TPI-Train)

The researchers built a massive new training camp called TPI-Train. Imagine this as a "School for Robot Butlers" designed specifically to teach them how to handle interruptions.

They created 88,000 practice scenarios. But here is the clever part: they didn't just give the robots easy examples. They created "Hard Negatives."

The Trick: They took a conversation where two people were talking, but they made the words sound like one person could have said them all.
The Lesson: The robot is forced to look at the sound of the voice (the pitch, the tone, the timbre) rather than just reading the words. It's like teaching a dog to distinguish between two people who are wearing the same outfit by listening to their footsteps, not by looking at their clothes.

The Test: The "Janus" Challenge

To see if the robots actually learned, the researchers created a special test called TPI-Bench, which includes a tricky part called the Janus-Test.

The Metaphor: Janus is the Roman god with two faces. In this test, the researchers take a conversation where two people are talking, but they re-record it so it sounds like one person speaking the whole time.
The Goal: If the robot is smart, it should say, "Wait, this sounds like two different people, even though the words flow together." If the robot is still confused, it will treat it as one person and fail the test.

The Results: From Confused to Confident

When they tested the old robots, they failed miserably. They kept mixing up the speakers.

But when they trained a robot using their new "School for Butlers" (TPI-Train) and the "Hard Negatives," the results were amazing:

It learned to listen to voices: The robot started paying attention to who was speaking, not just what they said.
It stopped guessing: Instead of guessing based on the words, it used the sound of the voice to know when to ignore a third person and when to listen to them.
It stayed polite: The robot learned to answer the main user while politely acknowledging the interruption if it was helpful, or ignoring it if it was just noise.

Why This Matters

This isn't just about fixing a bug; it's about making voice assistants feel more human. In real life, we talk in groups. We interrupt each other. We have side conversations.

By teaching robots to handle these "Third-Party Interruptions," the researchers are paving the way for voice assistants that can actually hang out with your whole family, understand who is talking to whom, and not get tripped up when the dog barks or your kid yells from the other room.

In short: They taught the robot to stop reading the script and start listening to the voices, so it never confuses your question with your friend's comment again.

1. Problem Statement

Current Spoken Language Models (SLMs) and Voice Assistants (VAs) excel in dyadic (one-on-one) interactions but fail significantly in Third-Party Interruption (TPI) scenarios.

The Core Failure: When a third party interrupts a primary user's query, current models often fail to distinguish the acoustic source. Instead of recognizing two distinct speakers, they treat the interruption as a continuous utterance or a "self-repair" by the primary user.
Consequences: This leads to "semantic shortcut learning," where models rely on textual coherence rather than acoustic cues (speaker changes). This results in:
- Contextual Collapse: Merging unrelated commands into nonsensical instructions.
- Security Risks: Executing unauthorized actions based on an interrupter's command (e.g., deleting data or making purchases).
- User Experience Degradation: Eroding trust and discouraging engagement with the VA.
The Gap: Existing datasets and benchmarks focus on single-speaker or dyadic interactions, lacking the complexity of triadic (User-VA-Third Party) dynamics and the specific acoustic discrimination required to handle interruptions.

2. Methodology

The authors propose a comprehensive framework spanning dataset construction, benchmarking, and model training to address TPI robustness.

A. Dataset Construction: TPI-Train

Scale: 88,000 instances covering 26 diverse TPI scenarios derived from canonical interruption taxonomies (e.g., Murata, Goldberg).
Scenario Types: Includes actionable interruptions (corrections, clarifications, constraints) and ignorable ones (off-topic remarks, social chatter).
Speaker-Aware Hard Negatives: To prevent semantic shortcut learning, the authors introduced a critical component: Hard Negatives.
- These are samples where the text of a two-speaker interruption is semantically coherent enough to look like a single-speaker monologue (e.g., self-correction).
- However, the audio is synthesized using only the primary speaker's voice.
- Goal: Force the model to rely on acoustic cues (speaker change detection) rather than semantic patterns to identify interruptions. If the text looks like one speaker but the audio has a second voice, the model must learn to detect the acoustic shift.
Response Strategy: The dataset is annotated with a flexible framework classifying interruptions as Actionable ( $C_A$ ) or Ignorable ( $C_I$ ).
- Actionable: The VA should acknowledge the third party and integrate their input (e.g., "The person with you suggested X, shall I proceed?").
- Ignorable: The VA should disregard the third party and answer the primary user.

B. Benchmarking: TPI-Bench

A rigorous evaluation framework consisting of three parts:

TPI-Test (2K samples): Evaluates Situation-Discriminative Response. Can the model generate the correct response strategy (Actionable vs. Ignorable) based on the scenario?
Janus-Test (2K samples): An adversarial test for Acoustic Sensitivity. It contains samples where the text implies a single speaker (semantically coherent), but the audio contains a second speaker.
- Metric: A good model should detect the interruption (low score on "single-speaker" assumption) despite the misleading text.
Real-World Test Set: 100 high-quality samples from real multi-party conversations (AMI, Friends-MMC, and human recordings) to validate transferability.

C. Evaluation Metrics

Response Strategy Following (RSF): Binary metric checking if the model followed the predefined strategy (Ignore vs. Acknowledge).
Overall Helpfulness (OH): A 5-point Likert scale (via LLM-as-a-judge and human evaluation) measuring naturalness and utility.
Auxiliary Metrics (BLEU/ROUGE-L): Measures similarity between responses to TPI-Test and Janus-Test. High similarity indicates the model is ignoring acoustic differences (shortcut learning).

3. Key Contributions

First TPI-Aware Framework: Defined TPI-awareness and provided an end-to-end pipeline from subjective strategy definition to dataset construction and evaluation.
TPI-Train Dataset: A large-scale dataset (88K) featuring speaker-aware hard negatives specifically designed to mitigate semantic shortcut learning and enforce acoustic grounding.
TPI-Bench: A comprehensive benchmark including the adversarial Janus-Test to isolate acoustic sensitivity from semantic understanding.
Reference Response Strategy: A validated framework for handling interruptions (Actionable vs. Ignorable) that aligns with human preferences.

4. Experimental Results

The authors fine-tuned Qwen2.5-Omni-7B using their framework and compared it against state-of-the-art baselines (ChatGPT-4o-audio, Kimi-Audio, VITA-Audio, Qwen3-Omni).

Baseline Performance: Existing models showed poor TPI awareness. They frequently conflated two speakers into one, resulting in low RSF scores and high BLEU/ROUGE similarity between TPI-Test and Janus-Test responses (indicating they ignored acoustic shifts).
Ablation Study:
- TPI-Base (trained only on TPI-Train): Improved TPI performance but suffered from domain overfitting, degrading general voice interaction capabilities.
- TPI-VA (Base + Dyadic data): Recovered general capabilities but still failed on Janus-Test, proving that simply balancing data distribution isn't enough; the model still relied on semantic shortcuts.
- TPI-Full (Base + Dyadic + Hard Negatives): Achieved the best results. It successfully distinguished speakers in the Janus-Test (low false positives) while maintaining high performance on general benchmarks (VoiceBench).
Quantitative Gains:
- TPI-Test RSF: Improved from ~0.24 (Baseline) to 0.83 (TPI-Full).
- Janus-Test RSF: Improved from ~0.12 (Baseline) to 0.16 (TPI-Full), indicating a massive reduction in misidentifying single-speaker audio as interruptions (or vice versa).
- Real-World Transfer: The model maintained significant gains on real-world data (RSF +0.43, OH +1.04), proving it did not overfit to synthetic patterns.
Embedding Visualization: t-SNE plots showed that the TPI-Full model created distinct clusters for single-speaker, two-speaker, and hard-negative inputs, whereas baselines showed heavy overlap.
Human Evaluation: The model's responses were preferred over baselines in 60.5% of A/B tests, and the generated responses aligned closely with human expectations for the reference strategy.

5. Significance

Overcoming Unimodal Reliance: The work demonstrates that SLMs can be trained to prioritize acoustic evidence over textual cues, a critical step toward robust multi-modal reasoning.
Safety and Usability: By preventing the "concatenation" of commands, the framework mitigates security risks (unauthorized actions) and improves user trust in real-world, multi-person environments.
Foundational Resource: TPI-Train and TPI-Bench provide the first standardized resources for developing and evaluating interruption-aware voice assistants, paving the way for more natural, full-duplex, multi-party spoken interactions.
Limitations & Future Work: The current work focuses on turn-based interactions. Future research aims to extend this to full-duplex systems where the assistant can interrupt or defer speaking in real-time, and to explore more diverse, user-specific response strategies.