Technical Summary: Towards Human-Like Interactive Speech Recognition with Agentic Correction and Semantic Evaluation

1. Problem Statement

Current Automatic Speech Recognition (ASR) systems predominantly operate under a single-pass paradigm, generating a transcription from an acoustic signal without the ability to incorporate user feedback for iterative refinement. This design contrasts sharply with natural human communication, where misunderstandings are routinely resolved through clarification, confirmation, and self-repair.

The paper identifies two critical mismatches in the current state of ASR:

1. Procedural Mismatch: Conventional systems treat follow-up utterances as new input rather than corrections to previous hypotheses. Once a meaning-critical error (e.g., misidentifying a named entity like "Megan" as "Morgan") occurs, the system lacks a mechanism to revise it within the same interaction session.
2. Evaluative Mismatch: Standard metrics like Word Error Rate (WER) and Character Error Rate (CER) quantify token-level mismatches but fail to distinguish between minor surface deviations (e.g., filler words) and errors that alter the intended meaning. In agent-oriented settings, a single error in a named entity can break downstream task execution, even if the WER is low.

2. Methodology

A. Task Formulation: Interactive ASR

The authors redefine ASR as a stateful multi-turn refinement task. Instead of independent one-shot decoding ($Y = ASR(I)$), the system maintains a transcription history $Y[:t-1]$ and updates the state at each turn $t$ based on new speech input $I_t$ and prior context:
$$Y_t = InteractiveASR(Y[:t-1], I_t)$$
This formulation enables the system to progressively repair meaning-critical errors through interaction.

B. Agentic ASR Framework

To implement this, the authors propose Agentic ASR, a closed-loop framework combining a single-pass ASR front-end with an LLM-based reasoning-and-editing module. The process at each turn involves three stages:

1. Semantic Correction: The raw ASR hypothesis ($H_t$) is mapped to an explicit, semantically coherent instruction ($H'_t$) by leveraging interaction history. This resolves inconsistencies in user correction utterances themselves.
2. Intent Routing: The instruction is classified into one of three intents:
   • Confirmation: The user accepts the current state.
   • New Input: The utterance is treated as new content.
   • Correction: The utterance is an edit to the existing state.
3. Reasoning-based Correction: If the intent is correction, a structured Locate–Reason–Modify pipeline is executed:
   • Locate: Identifies the specific span in the history to edit.
   • Reason: Infers the intended modification based on the instruction and context.
   • Modify: Applies the edit to produce the updated state.

C. Evaluation: Sentence-level Semantic Error Rate (S2ER)

To address the limitations of token-level metrics, the paper introduces S2ER, an LLM-based metric that evaluates whether a hypothesis preserves the intended meaning of the ground truth.

• Criterion: A binary judgment ($\hat{z} \in \{0, 1\}$) determining if the hypothesis is semantically equivalent to the reference, focusing on intent and key entities while ignoring disfluencies.
• Stability Protocol: To mitigate LLM judgment variance, the system employs a three-round bidirectional voting protocol. The judge evaluates the pair in both orders ($Y, Y_{GT}$ and $Y_{GT}, Y$) across three rounds; a positive label is assigned only if the majority of rounds confirm equivalence.

D. Interactive Simulation System (ISS)

To enable scalable, reproducible benchmarking without expensive human-in-the-loop testing, the authors developed the Interactive Simulation System (ISS).

• Mechanism: The ISS simulates multi-turn interactions. If the S2ER Judger determines the current transcription is not semantically equivalent to the ground truth, a User Simulator (comprising an LLM corrector and a TTS vocalizer) generates the next corrective spoken instruction.
• Termination: The loop continues until semantic equivalence is achieved or a maximum number of rounds is reached.

3. Key Contributions

1. Interactive ASR Task Formulation: Formalizing ASR as a stateful, multi-turn refinement process rather than independent single-pass decoding.
2. Agentic ASR Framework: A closed-loop architecture utilizing semantic correction, intent routing, and structured reasoning (Locate-Reason-Modify) to iteratively repair errors.
3. Semantic Evaluation System: The introduction of S2ER and the ISS, providing a reliable, scalable method to benchmark interactive ASR based on meaning preservation rather than lexical accuracy.

4. Experimental Results

Experiments were conducted on multilingual, named-entity-intensive, and code-switching benchmarks (including GigaSpeech, WenetSpeech, AISHELL-NER, and ASRU2019).

• Semantic Improvement: Iterative interaction consistently reduced S2ER across all benchmarks. For example, on the GigaSpeech Test set, S2ER dropped from 21.47% (Loop 0) to 3.49% (Loop 10). Similar significant gains were observed on named-entity and code-switching datasets.
• Discrepancy with Token Metrics: The gains in S2ER were substantially larger than improvements in conventional metrics (WER, CER, MER). This confirms that the framework primarily repairs meaning-critical errors rather than merely polishing surface-level token mismatches.
• Robustness to Backbone: The framework proved effective across various ASR backbones, including the weaker Whisper model. Even with a poor initial transcription (e.g., ~47% initial S2ER for Whisper), multi-turn interaction reduced errors to single digits, demonstrating that the interaction mechanism itself drives the improvement.
• LLM Scale: While smaller LLMs (8B) maintained the ability to improve semantic correctness, larger models (32B) provided more stable and precise edits, resulting in lower final error floors.
• Human-AI Alignment: The S2ER Judger showed high correlation (Pearson $r > 0.8$) with human expert judgments, validating the reliability of the automated semantic evaluation.

5. Significance and Claims

The paper argues that the mismatch between current single-pass ASR and natural human communication limits the reliability of speech interfaces, particularly for agents where intent preservation is critical.

The authors claim that Agentic ASR successfully bridges this gap by enabling iterative refinement, effectively recovering from meaning-critical errors that traditional systems cannot fix. Furthermore, they assert that S2ER is a necessary metric for this domain, as conventional token-level measures fail to capture the functional usability of the transcription in interactive settings. The work suggests that future ASR systems should move beyond one-shot prediction toward closed-loop, feedback-driven architectures, and that evaluation must shift from lexical accuracy to semantic equivalence.