💬 NLP

Interactive ASR: Towards Human-Like Interaction and Semantic Coherence Evaluation for Agentic Speech Recognition

This paper proposes an agentic framework for interactive speech recognition that leverages LLMs to both evaluate semantic coherence beyond traditional word error rates and enable human-like multi-turn correction, demonstrating improved fidelity across multiple languages and benchmarks.

Original authors: Peng Wang (X-LANCE Lab, Shanghai Jiao Tong University), Yanqiao Zhu (X-LANCE Lab, Shanghai Jiao Tong University), Zixuan Jiang (X-LANCE Lab, Shanghai Jiao Tong University), Qinyuan Chen (School of Com

Published 2026-04-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Peng Wang (X-LANCE Lab, Shanghai Jiao Tong University), Yanqiao Zhu (X-LANCE Lab, Shanghai Jiao Tong University), Zixuan Jiang (X-LANCE Lab, Shanghai Jiao Tong University), Qinyuan Chen (School of Computer Science, Fudan University), Xingjian Zhao (School of Computer Science, Fudan University), Xipeng Qiu (School of Computer Science, Fudan University), Wupeng Wang (Tongyi Fun Team, Alibaba Group), Zhifu Gao (Tongyi Fun Team, Alibaba Group), Xiangang Li (Tongyi Fun Team, Alibaba Group), Kai Yu (X-LANCE Lab, Shanghai Jiao Tong University), Xie Chen (X-LANCE Lab, Shanghai Jiao Tong University)

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to give a complex order to a very smart, but slightly hard-of-hearing, robot assistant.

The Old Way: The "Stuck Record" Problem
In the past, if you said, "Call Sarah Knight," and the robot heard, "Call Sarah Night," and you said, "No, Knight with a K!", the old robot would just stare at you blankly. It would say, "I heard 'Night'." It couldn't change its mind. It was like a record player stuck on a scratchy groove; once it played the wrong note, it couldn't go back and fix it.

Furthermore, the people who built these robots used a very strict ruler to grade them called WER (Word Error Rate). This ruler treated every mistake the same. If the robot got a tiny word like "the" wrong, it got the same "bad grade" as if it got a crucial name like "Knight" wrong. It was like a teacher giving you an 'F' for a spelling mistake on the word "the" in an essay, even if the rest of your story was perfect.

The New Way: The "Human-Like Conversation"
This paper introduces a new way of thinking called Interactive ASR (Automatic Speech Recognition). Think of this not as a robot that just listens, but as a collaborative editor that can chat with you to get things right.

Here is how it works, using a few simple analogies:

1. The New Grader: The "Meaning Detective"

Instead of using the old, rigid ruler (WER), the authors use a Meaning Detective (an AI called an LLM-as-a-Judge).

Old Ruler: Counts how many letters are wrong.
Meaning Detective: Reads the whole sentence and asks, "Did the robot understand what the user actually wanted?"
The Result: If the robot says "Night" instead of "Knight," but the context makes it clear you meant the person, the Detective gives it a pass. If the robot changes the meaning entirely, it gets a bad grade. This new score is called $S^2ER$ (Sentence-level Semantic Error Rate).

2. The New System: The "Surgical Editor"

The paper proposes a system that acts like a Surgical Editor.

Step 1 (The First Draft): You speak. The robot writes down what it thinks you said.
Step 2 (The Feedback): If you say, "No, that's wrong, it starts with a K," the robot doesn't just ignore you.
Step 3 (The Surgery): The robot uses a "Reasoning Brain" (powered by a large AI) to:
1. Locate the mistake (It finds "Night").
2. Reason about the fix (It thinks, "User said 'K', so 'Night' must be 'Knight'").
3. Surgically Replace the error (It swaps "Night" for "Knight" without rewriting the whole sentence).

3. The Simulation: The "Robot Acting as a Human"

To test this without needing thousands of real people to yell at robots, the authors built a Robot Simulator.

Imagine a robot that plays the role of a frustrated human user.
It listens to the first draft, realizes it's wrong, and then uses its own voice (Text-to-Speech) to say, "No, that's not right!"
The main robot then tries to fix it. They keep talking back and forth until the sentence is perfect.

Why Does This Matter?

The experiments showed that this new system is a game-changer:

Speed: In just one or two turns of conversation, the system fixes almost all the big mistakes. It's like having a conversation where you clarify things instantly, rather than getting stuck in a loop.
Accuracy: The "Meaning Detective" (the new metric) agrees with human experts 99% of the time, proving that checking for meaning is better than just counting words.
Realism: It works even when people switch between languages (like mixing English and Chinese) or speak with accents, because it focuses on the intent rather than just the sound.

In a Nutshell:
This paper teaches robots to stop being stubborn record players and start being helpful conversation partners. Instead of just saying "I heard this," they say, "I think you meant this, did I get it right?" and they are smart enough to fix their own mistakes when you tell them.

1. Problem Statement

The paper identifies two critical limitations in current Automatic Speech Recognition (ASR) systems:

Semantic-Blind Evaluation: The industry-standard metric, Word Error Rate (WER), treats all words equally. It fails to distinguish between errors in function words (low semantic impact) and critical content words like named entities (high semantic impact). Consequently, a low WER does not guarantee that the recognized text preserves the user's intended meaning.
Lack of Interactive Correction: Traditional ASR operates in a single-pass, static paradigm. Once an error occurs, the system cannot revise its output based on user feedback. In real-world scenarios (e.g., voice assistants), users often need to clarify or correct misrecognitions (e.g., "No, call Sarah Knight, not Night"), but current systems lack the mechanism to process these natural language corrections iteratively.

2. Methodology

The authors propose Interactive ASR, a framework that integrates an LLM-driven agent for correction and a new semantic evaluation metric.

A. The Interactive ASR Framework

The system moves beyond single-pass decoding ($Y = ASR(I)$) to an iterative, agentic process:

Base ASR: Generates an initial text hypothesis ( $H_t$ ) from user speech ( $I_t$ ).
LLM Intent Router: Analyzes the relationship between the new hypothesis and the previous transcript ( $Y_{t-1}$ $Y_{t - 1}$ ).
- If the input is a new utterance, it is output directly.
- If the input contains corrective intent, the system triggers the Reasoning Corrector.
Reasoning Corrector (Chain-of-Thought): An LLM agent refines the previous transcript ( $Y_{t-1}$ $Y_{t - 1}$ ) using the correction instruction ( $H_t$ $H_{t}$ ) via a three-step CoT process:
- Locate: Identify the specific erroneous segment in the previous transcript based on the user's instruction.
- Reason: Deduce the intended correction using phonetic or lexical constraints.
- Surgical Replacement: Replace only the erroneous segment while preserving the rest of the sentence structure.

B. Automated Simulation Framework

To evaluate this without expensive human-in-the-loop testing, the authors designed an automated simulation:

User Simulator: An LLM generates natural language correction instructions ( $C_t$ ) based on the ground truth vs. the ASR hypothesis. A zero-shot TTS model converts this text back into speech ( $I_t$ ) using the original audio as an acoustic reference to maintain speaker consistency.
Iterative Loop: The system processes the correction speech, updates the hypothesis, and repeats until the transcript matches the ground truth or a turn limit is reached.

C. New Evaluation Metric: $S^2ER$

The authors introduce Sentence-level Semantic Error Rate ( $S^2ER$ ).

Instead of counting token mismatches, an LLM-as-a-Judge evaluates whether the ASR output is semantically equivalent to the ground truth.
The judge prioritizes core intent and critical entities while ignoring minor surface variations (filler words, punctuation).
$S^2ER$ is calculated as the average rate of semantic mismatches across a dataset.

3. Key Contributions

$S^2ER$ Metric: A novel evaluation metric that leverages LLMs to assess semantic success rather than just token accuracy. The paper validates that LLM judgments correlate strongly (Pearson $r \approx 0.83$ ) with human consensus, often outperforming domain experts in consistency.
Agentic Correction Framework: A system architecture that enables ASR to accept spoken natural language feedback, reason about errors, and iteratively refine transcripts, mimicking human conversational repair.
Comprehensive Validation: Extensive experiments across three diverse benchmarks:
- GigaSpeech: Multi-domain English.
- WenetSpeech: Spontaneous Mandarin.
- ASRU 2019: Mandarin-English code-switching.

4. Experimental Results

Semantic Improvement: The framework significantly reduces $S^2ER$ $S^{2} E R$ .
- GigaSpeech: Dropped from 14.12% (Turn 0) to 6.03% (Turn 1) and 1.08% (Turn 10).
- WenetSpeech: Dropped from 15.56% to 6.26% (Turn 1) and 1.11% (Turn 10).
- ASRU 2019: Dropped from 26.89% to 8.10% (Turn 1) and 0.82% (Turn 10).
Rapid Convergence: The most significant improvements occur within the first 1–2 interaction turns, mirroring natural human patience limits.
Comparison with Traditional Metrics: While traditional metrics (WER, CER) showed only marginal improvements, $S^2ER$ showed drastic reductions, proving that the system successfully corrected meaningful errors even if the token-level changes were small.
Failure Analysis: Residual errors at high iteration counts (Turn 10) are primarily caused by the base ASR repeatedly misrecognizing the user's correction instructions, causing the LLM to lose the necessary anchors for surgical replacement.

5. Significance

Paradigm Shift: Moves ASR from a static, one-shot transcription task to a dynamic, interactive dialogue system capable of self-correction.
Evaluation Standard: Challenges the dominance of WER by establishing that semantic coherence is a more critical success metric for downstream tasks (e.g., command execution, intent understanding).
Human-Like Interaction: Demonstrates that LLM agents can effectively bridge the gap between ambiguous speech and precise intent, offering a path toward more robust and user-friendly voice interfaces.
Future Outlook: While current results rely on large models (Qwen3-32B), the framework establishes a theoretical upper bound for interactive ASR, guiding future research toward deploying these capabilities on smaller, more efficient architectures.