💬 NLP

RECOVER: Robust Entity Correction via agentic Orchestration of hypothesis Variants for Evidence-based Recovery

The paper introduces RECOVER, an agentic framework that leverages multiple ASR hypotheses and tool-using LLMs to robustly correct rare and domain-specific entity errors, achieving significant improvements in entity recall and word error rate across diverse datasets.

Original authors: Abhishek Kumar, Aashraya Sachdeva

Published 2026-03-18

📖 5 min read🧠 Deep dive

CC0 1.0

Original authors: Abhishek Kumar, Aashraya Sachdeva

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but your only witness is a very fast, very confident, but slightly hard-of-hearing assistant named Whisper.

Whisper listens to a conversation and writes down what they hear. Usually, they are great. But when it comes to specific, tricky names—like a rare medicine, a stock ticker, or a pilot's call sign—Whisper often gets it wrong. They might hear "cytiva" as "sitiva" or miss a whole word entirely. In fields like finance or medicine, getting these names wrong isn't just a typo; it's a disaster.

The paper introduces RECOVER, a new "super-detective" team designed to fix Whisper's mistakes without rewriting the whole story.

Here is how RECOVER works, broken down into simple steps with some creative analogies:

1. The "Multiple Guesses" Strategy (Multi-Hypothesis Generation)

Usually, Whisper gives you just one version of what was said. RECOVER is smarter. It asks Whisper to listen to the same audio five times, but each time, it tells Whisper to be a little bit "looser" or more creative.

The Analogy: Imagine asking five different friends to transcribe a mumbled sentence.
- Friend A hears "Cytiva."
- Friend B hears "Sitiva."
- Friend C hears "Cytiva" clearly.
- Friend D hears "Sativa."
- Friend E hears "Cytiva."

Even though they disagree, the fact that some of them got it right gives RECOVER a clue. If the correct answer appears in at least one of the guesses, RECOVER has a fighting chance to find it.

2. The "Wanted Poster" Search (Dynamic Entity Retrieval)

RECOVER knows exactly what names should be in the conversation (like a list of 6,000 medical drugs or 400 pilot call signs). It doesn't want to guess randomly; it wants to match the audio to the "Wanted Poster."

The Analogy: Think of this like a bouncer at a club. The bouncer has a list of VIPs. When someone tries to get in, the bouncer doesn't just look at their face; they check if the name sounds like a VIP, looks like a VIP, or is spelled similarly to a VIP.
RECOVER takes the five guesses from Whisper and scans them against the VIP list. It finds the top 200 most likely candidates that might be the correct name, even if they are spelled slightly wrong (e.g., matching "sitiva" to "cytiva").

3. The "Agentic Team" (The Three Tools)

This is the core of RECOVER. Instead of one robot doing everything, it uses a team of three specialized agents (tools) that work together like a production crew:

Tool 1: The Editor (Fuse Hypotheses)
- Job: Decides which of the five guesses is the best starting point.
- How: It looks at the "Wanted Poster" list. Which guess has the most correct names? It picks that one. Or, it might use a super-smart AI (LLM) to pick the best one.
- Analogy: A film editor choosing the best take from five different camera angles.
Tool 2: The Fixer (Propose Corrections)
- Job: Looks at the chosen text and says, "Hey, this word looks wrong. Let's swap it for the correct VIP name from our list."
- The Catch: This AI is very strict. It is only allowed to change specific names. It cannot rewrite the grammar, change the tone, or make up new words. It's like a spell-checker that only fixes proper nouns.
- Analogy: A surgeon who is only allowed to fix a specific broken bone and is forbidden from touching the rest of the body.
Tool 3: The Inspector (Verify & Apply)
- Job: Double-checks the Fixer's work before making the change permanent.
- The Rules:
  1. Is the new word actually on the VIP list?
  2. Did the Fixer just change the capitalization (e.g., "cytiva" to "Cytiva")? If so, ignore it.
  3. Is the new word actually similar to the old one? (You can't change "Star" to "Cytiva" just because you want to).
- Analogy: A quality control inspector at a factory. If the Fixer tries to swap a screw for a banana, the Inspector slaps the hand and says, "Nope, that's not allowed."

Why is this a big deal?

Most previous methods tried to fix errors by guessing or by using complex math inside the speech system itself. But RECOVER is special because:

It's a "Black Box" Fixer: It doesn't need to know how Whisper works inside. It just takes the output and fixes it. This means you can use it with any speech system.
It Prevents Hallucinations: AI often likes to make things up (hallucinate). By using the "Inspector" tool and strict rules, RECOVER ensures the AI only fixes things it is sure about.
It Saves the Day: In tests across finance, medicine, and air traffic control, RECOVER reduced errors in these tricky names by 8% to 46%. It found words that were completely missing in the original text and fixed them, all while keeping the rest of the conversation sounding natural.

The Bottom Line

RECOVER is like a highly skilled editor with a team of assistants. It listens to a messy transcript, checks it against a master list of important names, and carefully swaps out the wrong names for the right ones—without messing up the rest of the story. It turns a "good enough" transcription into a "perfect" one, specifically for the words that matter most.

1. Problem Statement

Automatic Speech Recognition (ASR) systems, even state-of-the-art models like Whisper, struggle with entity phrases (proper nouns, brand names, domain-specific terminology) that are rare or absent from training data.

The Challenge: In high-stakes domains (finance, medicine, air traffic control), entity errors are costly. If an entity is completely deleted or heavily corrupted in the primary (1-best) ASR output, standard post-processing or Large Language Model (LLM) correction often fails because the LLM lacks the necessary signal to recover the term.
Limitations of Existing Methods:
- Contextual Biasing: Requires access to decoder internals, which is often unavailable in black-box production ASR systems.
- Standard LLM Correction: Prone to hallucinations or over-correction without strict constraints.
- Evidence Gap: If the 1-best hypothesis misses an entity, the LLM cannot "guess" it correctly without additional evidence.

2. Methodology: The RECOVER Framework

RECOVER is an agentic correction framework designed as a tool-using agent. It bridges the evidence gap by leveraging multiple ASR hypotheses and applying constrained LLM editing. The pipeline consists of three main stages orchestrated by an agent:

A. Multi-Hypothesis Generation

Instead of relying on a single greedy decoding, the system generates $N$ transcript hypotheses from the same audio segment.

Implementation: In the experiments, five hypotheses ( $N=5$ ) are generated using Whisper-small with varying temperature sampling ( $T \in \{0.0, 0.2, 0.4, 0.6, 0.8\}$ ).
Goal: To create "complementary errors." If an entity is missed in one hypothesis, it is likely present (or less corrupted) in another.

B. Dynamic Entity Candidate Retrieval

To prevent LLM hallucinations and reduce context window waste, the system does not feed the entire entity list to the LLM.

Scoring Mechanism: It dynamically retrieves the top- $K$ $K$ (where $K=200$ $K = 200$ ) most relevant entity candidates for each segment using a three-signal scoring function:
1. Exact Token Hits: Count of exact word matches across all hypotheses.
2. Fuzzy Similarity: Normalized Levenshtein distance (weighted higher to catch near-misses like sitiva $\to$ cytiva).
3. Phonetic Prefix: Matching phonetic keys (e.g., 5-character prefix).
Output: A ranked list of top- $K$ candidates passed to the LLM.

C. Agentic Orchestration (Three Tools)

The agent manages three specialized tools to process the hypotheses and candidates:

Tool 1: Fuse Hypotheses
Combines the $N$ hypotheses into a single base transcript using one of four strategies:
- 1-Best: Uses the single greedy hypothesis ( $N=1$ ).
- Entity-Aware Select: Selects the hypothesis with the maximum number of exact entity-candidate matches.
- ROVER Ensemble: Aligns all hypotheses to a pivot (chosen via Entity-Aware Select) using Needleman–Wunsch alignment and performs a token-level majority vote.
- LLM-Select: Delegates the selection of the best base variant (and initial corrections) directly to the LLM based on the hypotheses and candidates.
Tool 2: Propose Corrections
A constrained LLM (e.g., GPT-4o) proposes find/replace edits strictly limited to the retrieved top- $K$ entity candidates.
- Constraints: No generic grammar rewrites; only entity substitutions (e.g., oscar kill papa $\to$ oscar kilo papa).
- Output: Valid JSON containing character offsets, original span, replacement span, and reasoning.
Tool 3: Verify & Apply
A deterministic guardrail layer that validates every proposed edit before application:
- Ensures the replacement exists in the entity list.
- Discards case-only changes.
- Relocates spans if offsets are incorrect.
- Enforces high Levenshtein similarity to prevent unrelated substitutions.
- Applies edits left-to-right, skipping overlaps.

3. Key Contributions

Multi-Hypothesis Diversity: Demonstrated that temperature sampling from a single ASR model generates complementary errors that significantly aid entity recovery.
Agentic Architecture: Introduced a modular pipeline (Fuse $\to$ Propose $\to$ Verify) that separates analysis, correction, and safety verification.
Four Correction Strategies: Evaluated 1-Best, Entity-Aware Select, ROVER Ensemble, and LLM-Select, showing that LLM-Select offers the best balance of performance and safety.
Constrained Editing: Proved that strict constraints (entity lists + deterministic guardrails) allow LLMs to correct entities effectively without hallucinating new content.

4. Experimental Results

The framework was evaluated across five diverse datasets: Earnings-21 (Finance), ATCO2 (Air Traffic Control), Eka-Medical (Medical), Common Voice (General), and ContextASR-Bench (Dialogue/Movies).

Performance Gains:
- Achieved 8% to 46% relative reduction in Entity-Phrase Word Error Rate (E-WER).
- Increased Entity Recall by up to 22 percentage points.
- LLM-Select was the most consistent strategy, achieving the best E-WER on 4 out of 5 datasets while maintaining or improving overall WER.
Specific Findings:
- Recall vs. Precision: The primary driver of improvement was fixing deletions and substitutions (near-misses). Precision remained largely preserved or traded off minimally.
- Robustness: While ROVER Ensemble and Entity-Aware Select sometimes degraded overall WER in noisy domains (due to insertion noise from token merging), LLM-Select avoided this by selecting the best single variant rather than merging tokens.
- Ablation: Using a smaller LLM (GPT-4o-mini) still yielded consistent gains (2.2–15.7% RWERR), though a stronger LLM (GPT-4o) provided superior reasoning for complex near-misses.

5. Significance

Black-Box Compatibility: Unlike contextual biasing, RECOVER works with production ASR systems as black boxes, requiring no access to decoder internals.
Safety-Critical Application: The deterministic guardrails make it suitable for domains where hallucinations are unacceptable (e.g., medicine, aviation).
Paradigm Shift: Moves post-ASR correction from simple text refinement to an evidence-based agentic process, utilizing the diversity of multiple hypotheses to recover missing information that a single transcript cannot provide.
Scalability: The framework is agnostic to the hypothesis generation method and can be adapted to any ASR model or entity list.

In conclusion, RECOVER demonstrates that combining multi-hypothesis diversity with constrained, agentic LLM editing is a highly effective strategy for recovering rare and critical entity phrases in speech recognition, significantly outperforming baseline methods without compromising overall transcript quality.