💬 NLP

EchoChain: A Full-Duplex Benchmark for State-Update Reasoning Under Interruptions

The paper introduces EchoChain, a new benchmark designed to evaluate and diagnose the significant failures of real-time voice assistants in updating task states when interrupted mid-speech, revealing that current systems struggle with contextual inertia, interruption amnesia, and objective displacement.

Original authors: Smit Nautambhai Modi, Gandharv Mahajan, Marc Wetter, Randall Welles

Published 2026-04-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Smit Nautambhai Modi, Gandharv Mahajan, Marc Wetter, Randall Welles

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are having a lively conversation with a very smart, but slightly clumsy, friend. You are in the middle of telling them a story about your vacation, and just as you're describing the beach, they suddenly jump in and say, "Wait! I just remembered, I'm allergic to shellfish, so don't recommend any seafood!"

A perfect friend would immediately stop talking about the beach, apologize, and say, "Oh no! Okay, let's talk about the mountains instead."

But what if your friend said, "Oh, got it, no shellfish," and then kept talking about the beach? Or what if they said, "No shellfish, mountains it is!" for a second, but then five seconds later started talking about seafood again? Or what if they got so excited about your shellfish allergy that they forgot you were even on vacation and started talking about their own diet?

This paper, EchoChain, is basically a report card for AI voice assistants on how well they handle these kinds of "mid-sentence interruptions."

The Problem: The "Half-Duplex" Habit

Most AI voice assistants today are trained like old-school walkie-talkies. They operate in half-duplex mode: You talk, they listen. Then, they stop talking, and you talk again. They wait for you to hit "Enter" (or stop speaking) before they process anything new.

But real humans are full-duplex. We interrupt, we say "wait," we add "oh, and one more thing" while the other person is still talking. Current AI assistants are terrible at this. When you interrupt them, they often:

Pretend they didn't hear you.
Forget what you just said a moment later.
Get so distracted by your interruption that they forget the original task.

The Solution: EchoChain (The "Interruption Gym")

The researchers built a special testing ground called EchoChain. Think of it as a gym specifically designed to train AI to handle interruptions.

Instead of just asking the AI a question and waiting for an answer, EchoChain:

Synthesizes human voices: It uses realistic AI voices to talk to the assistants being tested.
Times the interruptions perfectly: It waits for the exact moment the AI starts speaking and then "barges in" with a new piece of information (like the shellfish allergy example).
Measures the reaction: It checks if the AI corrected its path or if it stumbled.

The Three Ways AI Fails (The "Failure Taxonomy")

The paper found that when AI gets interrupted, it usually fails in one of three funny (but frustrating) ways:

Contextual Inertia (The "Stubborn Mule"):
- What happens: The AI hears you say "No shellfish," nods along, says "Got it," but then keeps recommending shrimp.
- Analogy: It's like a GPS that says, "Okay, traffic is bad, let's take a detour," but then keeps driving you straight into the traffic jam anyway. It acknowledges the change but refuses to actually change its mind.
Interruption Amnesia (The "Goldfish"):
- What happens: The AI hears you, updates its plan correctly for a few seconds, but then suddenly forgets the new rule and reverts to the old plan.
- Analogy: Imagine a chef who hears you say "No onions!" and starts chopping celery. But then, three seconds later, they reach for the onion bowl again and say, "Here's your soup!" They forgot the instruction almost immediately.
Objective Displacement (The "Distraction Magnet"):
- What happens: The AI gets so focused on your interruption that it completely drops the original task.
- Analogy: You ask the AI to "Plan a dinner menu." You interrupt with "Oh, by the way, I love Italian food." The AI then spends the next 10 minutes talking only about Italian food and forgets to actually give you a menu. It lost the plot entirely.

The Results: The AI is Still Learning

The researchers tested four of the smartest, most advanced voice AI models available today. The results were sobering:

No model passed 50% of the time. Even the best one failed more than half the time when interrupted mid-sentence.
The "Interruption" is the real problem: When they tested the same tasks without interruptions, the AI did much better. This proves the AI isn't just "bad at the task"; it specifically breaks down when it has to think and talk at the same time.

Why This Matters

We are moving toward a future where we talk to AI like we talk to humans—naturally, with interruptions and overlapping speech. If our voice assistants can't handle a simple "Wait, I changed my mind" while they are talking, they will always feel robotic and frustrating.

EchoChain gives scientists a way to measure exactly how and why these assistants fail, so they can build models that are truly "full-duplex"—smart enough to listen, think, and change their minds, all while keeping the conversation flowing.

1. Problem Statement

Current spoken dialogue systems (SDS) are predominantly evaluated using half-duplex benchmarks, which assume a strict turn-taking protocol where a user speaks, the system processes the entire utterance, and then responds. However, human communication is naturally full-duplex: users frequently interrupt, revise constraints, or add new information while the assistant is mid-generation.

Existing benchmarks fail to evaluate how models handle state-update reasoning during these interruptions. When a user interrupts a model's response, the model must:

Acknowledge the new input.
Revise its in-progress reasoning.
Maintain the original task context while integrating new constraints.

Current systems often fail silently, producing fluent responses that ignore the interruption, forget previous constraints, or abandon the original task entirely. There is a lack of standardized, controlled benchmarks to diagnose these specific failure modes.

2. Methodology: The EchoChain Framework

The authors introduce EchoChain, a controlled benchmark designed to evaluate full-duplex state-update reasoning. The framework addresses the difficulty of comparing models by normalizing acoustic and temporal variables.

A. Architecture and Components

The framework operates via a Planner-Simulator Loop:

Dialogue Planner (GPT-5.2): Generates a conversation blueprint based on scenario seeds (e.g., voice support, interview coaching). It steers the interaction toward states where interruptions are contextually appropriate.
User Simulator (Gemini 3 Pro): Generates user messages and barge-in (interruption) text conditioned on the current blueprint and conversation history.
Persona Voice Engine: Synthesizes all user audio before the interaction begins using voice cloning (Minimax). This ensures acoustic latency is identical across all tested models, eliminating variable synthesis delays as a confounding factor.
Full-Duplex Orchestrator: Streams the pre-synthesized audio to the target Realtime Responder Model. It monitors for the assistant's speech onset and injects the barge-in stimulus at a fixed offset relative to that onset. This standardizes the timing of the interruption for every model.
Evaluation Layer: Uses an automated Failure Classifier (GPT-5.2) followed by human review to adjudicate pass/fail outcomes based on a specific taxonomy.

B. Experimental Controls

To ensure fair cross-model comparison, EchoChain controls three key variables:

Acoustic Latency: Pre-synthesized audio ensures identical input timing.
Interruption Timing: Barge-ins occur at a standardized offset from the assistant's speech start.
Contextual Consistency: System instructions, personas, and conversation history are held constant across models.

C. Validation of Synthetic Audio

The authors conducted a paired listening study (240 judgments) comparing cloned voices against real human recordings. A Pearson goodness-of-fit test ( $X^2 = 0.92, p = 0.63$ ) showed no significant directional preference, validating that the synthetic voices are a realistic proxy for human speech in this benchmark.

3. Key Contributions: Failure Taxonomy

The paper defines a diagnostic taxonomy for post-interruption failures, categorizing them into three distinct patterns:

Contextual Inertia: The model acknowledges the interruption but fails to integrate the new information into the actual content. It continues generating as if the interruption never happened or applies the update incorrectly (e.g., acknowledging a "no caffeine" constraint but still recommending coffee).
Interruption Amnesia: The model initially incorporates the new constraint correctly but "forgets" it as generation proceeds, reverting to the original plan or dropping the constraint in subsequent sentences.
Objective Displacement: The model pivots entirely to the interruption, abandoning the original task. Instead of integrating the new info into the original goal, it treats the interruption as a new, separate request, leaving the original task incomplete.

4. Results

The benchmark was evaluated on four state-of-the-art full-duplex voice models: GPT-realtime, Gemini Live, Grok Voice Agent, and Amazon Nova Sonic 2.

Overall Performance: No model exceeded a 50% Mean Pass Rate (MPR) across 200 interrupted conversations.
- Grok Voice Agent: Highest MPR at 48.5%.
- GPT-realtime: 45.0% MPR.
- Gemini Live: 16.5% MPR.
- Amazon Nova Sonic 2: 26.5% MPR.
Failure Distribution:
- Gemini Live failed primarily due to Interruption Amnesia and Objective Displacement.
- Amazon Nova Sonic 2 failed mostly due to Contextual Inertia (ignoring/misapplying updates).
- Grok showed the strongest retention of original objectives (lowest Objective Displacement).
Control Experiment: A paired half-duplex control (where the same content was delivered after the assistant finished speaking) showed a 40.22% reduction in total failures. This confirms that the majority of errors are driven specifically by the interruption event and the need for mid-generation state revision, rather than general task difficulty.

5. Significance and Impact

Diagnostic Precision: EchoChain moves beyond aggregate task scores to provide granular insights into how models fail (e.g., memory vs. integration vs. task abandonment). This allows developers to target specific architectural weaknesses.
Benchmarking Standard: It establishes the first reproducible, controlled benchmark for full-duplex state-update reasoning, isolating the variable of "mid-generation interruption" from acoustic latency and context drift.
Future Direction: The results indicate that current full-duplex systems are not yet ready for robust real-world deployment where users frequently interrupt. The benchmark provides a baseline for tracking progress as models improve their ability to revise in-progress reasoning under concurrent input.

In conclusion, EchoChain demonstrates that while current models can handle turn-based dialogue, they struggle significantly with the dynamic, real-time state revision required for natural, full-duplex human-AI interaction.