Technical Summary: IHBench – Evaluating Post-Interruption Recovery in Voice Agents

Problem Statement

Real-time voice agents are increasingly deployed in structured, multi-step workflows (e.g., customer service, healthcare scheduling). While natural spoken conversation is defined by interruptions—users cutting in to correct information, express impatience, or provide backchannels—existing benchmarks primarily evaluate the mechanics of interruption (e.g., barge-in detection, endpointing, turn-taking timing).

A critical gap remains: What happens after the interruption? Current benchmarks do not measure whether an agent successfully recovers the workflow state, addresses the user's interjection, or avoids re-delivering content the user already heard. Failure in this "post-interruption recovery" leads to qualitatively broken interactions (e.g., an agent repeating a sentence the user just corrected, or losing its place in a state machine), which is distinct from merely failing to stop speaking.

Methodology: IHBench Design

The authors introduce IHBENCH (Interruption Handling Benchmark), a synthetic benchmark designed to evaluate post-interruption recovery in voice agents executing state-machine-driven workflows.

1. Data Generation Pipeline

The benchmark consists of 45 synthetically generated conversations across 10 enterprise domains (SaaS, finance, healthcare, etc.). The pipeline involves:

• Workflow Grounding: Each conversation is driven by a strict state machine (e.g., Authenticate → Verify → Process → Confirm).
• Controlled Interruptions: Interruptions are injected at specific, controlled points mid-utterance (not just at turn boundaries) by a user simulator.
• Six Interruption Types: The benchmark defines six distinct interruption intents, each requiring different recovery behaviors:
  • Normal: Cooperative cut-in (new detail/constraint).
  • Impatient: User wants to skip explanations.
  • Correction: User corrects previously stated information.
  • Topic Switch: User introduces an unrelated request.
  • Filler: Brief backchannel (e.g., "mm-hm") requiring seamless continuation.
  • Pushback: User resists a claim or request.
• Rubric Generation: For every interruption, a specific evaluation rubric is generated before model inference, defining task fulfillment goals and recovery quality criteria.
• Verification: A verifier ensures no state-machine leakage (users knowing internal workflow stages) and checks for naturalness.

2. Evaluation Metrics

Each interruption point serves as an evaluation sample scored on two independent axes by LLM judges (GPT-5.4-mini):

1. Task Fulfillment (Comparative): The model's response is compared against a baseline (GPT-4o Audio) response. A judge selects the winner based on which response better advances the workflow goal. The metric is the Win Rate (0.50 = parity with baseline).
2. Recovery Quality (Absolute): The model's response is evaluated against 2–4 type-specific criteria (e.g., "did it continue exactly from the cutoff point?" for fillers). The metric is the Pass Rate (fraction of interruptions where all criteria are met).

3. Experimental Setup

• Models Evaluated: 27 configurations from OpenAI (GPT-4o, GPT Realtime, GPT Audio), Google (Gemini 2.5/3.x), and open-weight models (Gemma, Qwen, MiMo, Kimi, etc.).
• Input Modalities: Evaluated on both audio input and text-only transcripts to isolate modality effects.
• Validation: Agreement studies were conducted using a secondary judge (Gemini 3 Flash) and human annotators (Prolific) to validate the LLM-as-a-judge approach.

Key Results

1. Closed-Weight vs. Open-Weight Gap

Closed-weight models consistently outperform open-weight models across all metrics:

• Task Fulfillment: Closed-weight models win significantly more often against the baseline.
• Depth Degradation: As conversation depth increases, open-weight models degrade roughly 3.3× faster than closed-weight models. The mean slope for open-weight models is -0.053 per turn, compared to -0.016 for closed-weight models.
• Modality Gap: Closed-weight models (specifically Gemini) show no significant performance gap between audio and text inputs. Open-weight models perform significantly worse on audio than text (approx. 8 points lower on Task Fulfillment).

2. Distinct Capability Axes

The study identifies that Task Fulfillment and Recovery Quality are partially independent axes:

• The model with the highest Task Fulfillment (GPT Realtime 2) is only mid-tier in Recovery Quality.
• The model with the highest Recovery Quality (Gemini 2.5 Flash) has lower Task Fulfillment.
• Cross-Benchmark Analysis: When compared against AudioMultiChallenge (AMC), IHBench's Recovery Quality metric shows the lowest correlation with existing multi-turn audio benchmarks. This suggests recovery quality is a distinct capability not captured by general conversational benchmarks.

3. Interruption Type Sensitivity

Performance varies drastically by interruption type:

• Filler Handling: This is the strongest differentiator. GPT-family models fail to continue an utterance correctly after a backchannel only 7%–31% of the time. In contrast, the Gemini 2.5 family achieves 62%–68%, while the newer Gemini 3.x line regresses sharply to 13%–32%.
• Normal/Topic Switch: These are generally handled well across most models.
• Correction/Impatient: These show moderate difficulty, with significant variance between models.

4. Validation

• Judge Agreement: The primary LLM judge agrees with a secondary judge (Cohen's κ ≈ 0.75) and human annotators (κ ≈ 0.51) at levels comparable to or exceeding human inter-annotator agreement.
• Ranking Stability: Human rankings of models match the LLM judge's rankings almost exactly (Spearman ρ = 1.0 for Recovery Quality).

Significance and Claims

The paper claims that post-interruption recovery is a distinct and critical capability for voice agents that is currently under-evaluated. The authors argue that:

1. Current Benchmarks are Insufficient: Existing benchmarks focus on when to stop speaking, not what to say next to maintain workflow integrity.
2. Targeted Evaluation Reveals Training Gaps: The large variance in performance (especially on filler handling) suggests these behaviors are undertrained rather than inherently difficult, implying that targeted evaluation can guide specific training improvements.
3. Modality Matters for Open Models: Open-weight models struggle significantly more with audio inputs than text, indicating a need for better audio-text alignment in these architectures.
4. Recovery is a New Axis: Recovery quality is not merely a subset of general conversational ability; it is a unique dimension of voice agent performance that requires specific benchmarking.

The authors conclude that IHBench provides a necessary framework for measuring and improving the robustness of voice agents in real-world, interruption-heavy environments, highlighting a clear divide between current frontier closed-weight models and open-weight alternatives.