💻 computer science

AgenticASR: Refining Speech Recognition in Real-World Scenarios via an Agentic Approach

The paper introduces AgenticASR, an agentic framework that refines real-time speech recognition by iteratively revising transcripts to remove disfluencies and resolve self-corrections while preserving the speaker's final intent, validated by a new bilingual benchmark and superior performance across multiple ASR front ends.

Original authors: Zixuan Jiang, Binghao Qiang, Jiaying Chi, Yanqiao Zhu, Kai Yu, Xie Chen

Published 2026-07-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zixuan Jiang, Binghao Qiang, Jiaying Chi, Yanqiao Zhu, Kai Yu, Xie Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to write down a friend's story, but your friend talks exactly like a human: they stumble over words, say "um" and "uh," start a sentence, realize they made a mistake, and then correct themselves mid-stream. If you write down every single sound they make, you get a messy, confusing transcript full of false starts and repetitions. This is what traditional speech-to-text technology does; it's a very literal scribe that captures the sound of speech but not the intent of the message. On the other hand, some newer tools try to clean up the text, but they usually wait until the person has finished speaking entirely before they start editing. This creates a problem: if your friend says, "Meet me at Mary's house... wait, no, Marie's house," a tool that waits until the end might have already written "Mary" and can't go back to fix it without starting over. The goal of this research is to build a system that acts like a super-fast, attentive editor who listens in real-time, catches mistakes as they happen, and instantly rewrites the text to be clean and readable, all while the person is still talking.

The paper introduces a new approach called AgenticASR (Agentic Speech Recognition). Think of it as a two-person team working on a live broadcast. The first person is the "Scribe" (the ASR front end), who quickly types down exactly what they hear, including all the stammers and "ums." The second person is the "Refiner" (the Agentic part), a smart editor who constantly glances at the Scribe's latest few sentences. As soon as the Scribe types a chunk of text, the Refiner looks at it, fixes the mess, and swaps the messy version for a clean one.

Here is the magic trick: the Refiner doesn't just look at the current sentence; it keeps a small "window" of the last few chunks of text in its mind. If the Scribe types "Meet Mary," and then the next chunk comes in as "Wait, Marie," the Refiner sees the connection immediately. It realizes the first chunk was a mistake, deletes "Mary," and updates the whole sentence to "Meet Marie" before the speaker even finishes the next thought. This allows the system to handle self-corrections and explanations on the fly, rather than waiting for the speech to stop.

To test if this actually works, the authors built a special playground called AASR-Bench. Imagine a giant obstacle course with 917 different scenarios, ranging from casual chats and customer service calls to technical coding sessions and voice searches. They created 6,637 tiny, specific questions (rubrics) to grade the results, checking things like: "Did they remove the 'ums'?", "Did they fix the spelling of the name?", and "Did they keep the final meaning correct?" They didn't just look at how many words were right; they looked at how readable and useful the text was.

The results suggest that this "Agentic" approach is a significant step forward. When they tested AgenticASR against other systems, it consistently produced cleaner, more accurate text. For example, using a specific setup (Qwen3-ASR-1.7B with their Refiner), the system scored 79.95 on their overall benchmark, beating the next best methods by a comfortable margin. The study found that the system works best when it looks at a "window" of about three chunks of speech at a time. This window size was the sweet spot: it was large enough to catch corrections that happened a few seconds ago (like the "Mary" to "Marie" switch) but small enough to keep the system fast.

The authors also discovered that the size of the "Refiner" matters. A bigger, smarter editor (a 4-billion parameter model) did a slightly better job at fixing complex sentences than a smaller one, but it took a bit longer to think. However, even the smaller, faster models were much better than systems that just waited until the end to edit. The paper emphasizes that this system is not perfect; if the initial Scribe misses a word entirely, the Refiner can't magically invent it back. But for the messy, real-world stuff we actually say—fillers, stutters, and mid-sentence corrections—AgenticASR suggests a practical way to get clean, readable text instantly, turning the chaotic stream of human speech into a polished story as it happens.

Technical Summary: AgenticASR

Problem Definition

Automatic Speech Recognition (ASR) has achieved high accuracy in verbatim transcription, yet raw transcripts often fail to produce immediately usable text. Spontaneous speech contains disfluencies (fillers, repetitions, false starts), self-corrections, and spoken-form expressions that obscure the speaker's final intent and increase reading effort. While existing spoken-to-written methods exist, they generally operate in two suboptimal modes:

Offline Post-Transformation: Systems process completed audio or transcripts, delaying clean text generation until an utterance ends. This prevents the system from revising earlier text when later speech (e.g., a spelling correction or explanation) changes the interpretation of preceding content.
Early Emission without Revision: Emitting cleaned partial hypotheses as speech arrives is insufficient because subsequent corrections can invalidate previously committed text.

The core challenge is to perform Agentic Speech Recognition (AgenticSR): an audio-to-clean-text task that removes disfluencies, resolves self-corrections, and normalizes written form while preserving the speaker's final intended message, all within an online, streaming context where the system must revise its own output as new evidence arrives.

Methodology

The authors propose AgenticASR, a two-stage system designed to implement AgenticSR through an ASR–Refiner architecture.

1. System Architecture

ASR Front End: Generates an intermediate spoken-form hypothesis (which may contain errors and disfluencies).
Refiner: A language model (LM) that transforms the ASR hypothesis into clean written text.
Online Inference Mechanism: Unlike traditional streaming ASR that emits tokens sequentially, AgenticASR employs a sliding window approach.
- A Chunk Manager identifies stable source-text spans using Voice Activity Detection (VAD) and punctuation.
- At each update $t$ , the system concatenates the current source span with up to $K-1$ preceding spans (default $K=3$ ).
- The Refiner processes this bounded context to generate a single refined string.
- The system replaces the corresponding local span in the emitted transcript with this refined string.
- This allows later speech to revise earlier output (e.g., correcting a name from "Mary" to "Marie" after a spelling explanation) without reprocessing the entire stream.

2. Data Generation Pipeline

To train the compact Refiner, the authors developed an LLM-assisted data-generation pipeline that constructs training pairs without requiring new audio recordings:

Seed Generation: LLMs generate pools of entities, terms, and patterns for 10 usage scenes (e.g., academic, customer service, vibe coding).
Oral Generation: LLMs compose coherent oral utterances from seeds, introducing controlled disfluencies, repetitions, and four types of correction structures (no correction, single, rollback, multiple).
Clean Generation: LLMs transform the oral utterances into clean written targets, resolving self-corrections and applying Inverse Text Normalization (ITN).
ASR Simulation: LLMs simulate ASR errors (truncation, word omissions, recognition errors) on the oral input to create realistic $(H_i, Y_i)$ pairs where $H_i$ is the noisy ASR hypothesis and $Y_i$ is the clean target.
Quality Control: Automated and manual checks ensure semantic preservation and remove near-duplicates.

3. Evaluation Benchmark: AASR-Bench

The authors introduce AASR-Bench, the first dedicated bilingual benchmark for AgenticSR.

Composition: 917 utterances across 10 usage scenes, totaling 4.2 hours of audio.
Evaluation Protocol: Instead of relying solely on token-level metrics (WER/CER), the benchmark uses 6,637 atomic rubrics generated by an LLM (Qwen3.7-Plus) and judged by another LLM (Gemma-4-31B-IT).
Rubric Dimensions:
- Content: Preservation of semantic units unaffected by disfluencies.
- Format: Correct ITN and written formatting of numbers, dates, and entities.
- Filter: Removal of semantically empty fillers and repetitions.
- Rephrase: Resolution of self-corrections and multi-stage revisions to the final intent.
Scoring: A weighted micro-average of rubric scores (ranging from -1 to 2 depending on the dimension) to penalize hallucinations and reward faithful transformation.

Key Contributions

Definition of AgenticSR: Formalizing the task of final-intent-preserving audio-to-clean-text recognition that supports continual emission and revision.
AASR-Bench: A bilingual benchmark with fine-grained, rubric-based evaluation that captures semantic quality, formatting, and correction resolution better than traditional WER.
AgenticASR System: A decoupled ASR-Refiner framework capable of online revision via a bounded sliding window, enabling the correction of earlier output based on later evidence.
Data Pipeline: An LLM-assisted method for generating high-quality training pairs for the Refiner, decoupling the training data from the need for massive manually annotated audio datasets.

Experimental Results

Experiments were conducted using various ASR front ends (Qwen3-ASR, Whisper) and Refiner models (MiniCPM-5-1B, Qwen2.5 variants).

Performance: AgenticASR achieved the highest Overall scores on AASR-Bench across all evaluated ASR backends.
- With Qwen3-ASR-1.7B, AgenticASR scored 79.95, outperforming the API-based transformation baseline (69.93) by 10.02 points and the direct speech-to-clean-text model (FormalASR) by 27.45 points.
- Improvements were most significant in Format (normalization) and Rephrase (correction resolution), dimensions where traditional metrics often fail.
Online vs. Offline: A three-chunk window ( $K=3$ ) in online inference recovered nearly all the quality of offline inference (Rephrase score 70.47 vs. 72.83 offline) with only a modest latency increase (12.15s vs. 9.59s).
Refiner Capacity: Larger Refiners (4B) improved performance further but increased latency, suggesting a trade-off between quality and real-time constraints.
Human Agreement: A study with domain experts showed strong agreement (Spearman $\rho \approx 0.82$ ) between the LLM-based rubric judge and human assessments, validating the evaluation protocol.
Dependency on ASR Quality: The system's performance is tightly coupled to the upstream ASR. While AgenticASR improved results across all Whisper scales, systems with weaker ASR front ends (Whisper Base) showed lower Content preservation scores, indicating the Refiner cannot recover content lost by the initial recognizer.

Significance and Claims

The paper positions AgenticASR as a practical framework for intent-preserving clean transcription during ongoing speech. Its primary significance lies in bridging the gap between verbatim transcription and readable text in real-time scenarios.

Practicality: By using a bounded context window, the system avoids the computational cost of reprocessing entire streams while still allowing for the revision of earlier text when later speech provides disambiguation (e.g., spelling corrections).
Evaluation Shift: The work argues that token-level metrics are insufficient for spoken-to-written tasks and demonstrates the utility of rubric-based, semantic evaluation.
Limitations: The authors modestly note that the system's effectiveness is bounded by the quality of the upstream ASR; if the initial recognizer fails to capture the semantic content, the Refiner cannot reconstruct it. Additionally, the current implementation relies on a fixed window size, which may not capture very long-range dependencies.

The authors conclude that this approach is well-suited for voice assistants, meeting transcription, and live dictation, where readable text is required before an utterance is fully complete.