How to Recognize New Words: A Comparison Between Context Biasing Methods and Speech LLMs

Technical Summary: Recognizing New Words – Context Biasing vs. Speech LLMs

Problem Statement
Automatic Speech Recognition (ASR) systems, particularly those based on end-to-end transformer architectures like Whisper, achieve low overall Word Error Rates (WER) across diverse conditions. However, they exhibit a persistent weakness in recognizing rare or absent words from training data, such as named entities, acronyms, and domain-specific terminology. Because these words are often the most informative parts of an utterance, errors on them disproportionately degrade transcript utility, even if the overall WER remains low due to the dilution of these errors among frequent function words.

Methodology
The paper presents a controlled, head-to-head comparison of two distinct strategies for addressing this challenge:

1. Context Biasing Methods: These extend a base ASR model (Whisper-large-v2) to accept a list of words or phrases at inference without altering model parameters for each new list. The authors implement and evaluate two specific architectures based on prior work:
   • Method A: Uses a dedicated context-attention layer within the decoder. It predicts the availability of relevant information in the bias list and extracts the most relevant item using attention mechanisms.
   • Method B: Extends the decoder's vocabulary by adding dynamic tokens corresponding to the bias list entries. The model learns to replace subsequences in the output with these dynamic tokens, which are embedded based on the context list.
2. Speech Large Language Models (Speech LLMs): Three recent models (Qwen3-ASR, Qwen3-Omni, and VibeVoice-ASR) that jointly consume audio and text prompts. In this setup, the words of interest are placed directly into the text prompt, allowing the model to attend to them during generation.

The evaluation spans three datasets: Earnings-21 (non-read, domain-specific), LibriSpeech (read speech, split into test-clean and test-other), and Yodas (non-read, YouTube data). The study systematically probes robustness by varying the number of distractors (irrelevant words added to the list) and analyzing sensitivity to the order of words in the prompt. Metrics include Biased WER (BWER), Unbiased WER (UWER), and overall WER.

Key Contributions

• Comparative Evaluation: A systematic comparison of Whisper-based context biasing against three state-of-the-art speech LLMs across read and non-read speech scenarios.
• Robustness Analysis: Quantification of sensitivity to distractor counts and prompt word ordering, revealing significant differences in how the two approaches handle noisy context.
• Filtering Mechanism: An investigation into using an auxiliary speech LLM to filter the bias list before inference, aiming to mitigate the sensitivity of speech LLMs to long lists of distractors.
• Trade-off Characterization: A clear delineation of the performance trade-offs between dedicated biasing methods and flexible speech LLMs to guide method selection.

Results

• Context Biasing Performance: Both biasing methods significantly reduce BWER (up to 70% relative for Method A and 88% for Method B) while leaving UWER largely unaffected. Method A demonstrates high robustness to distractors, with performance degradation of only up to 2% (BWER) and 7% (UWER) when the correct word is present, but up to 17% (BWER) and 12% (UWER) when it is absent. Method B achieves lower BWER than A in clean conditions but suffers more significantly when distractors are added; when the correct word is present, degradation reaches up to 17% (BWER) and 19% (UWER), while when the correct word is absent, degradation reaches up to 55% (BWER) and 20% (UWER), primarily due to insertion errors from wrongly predicted dynamic tokens.
• Speech LLM Performance: Speech LLMs excel on read speech (LibriSpeech), with Qwen3-Omni achieving the lowest BWER in distractor-free settings. However, they generalize poorly to non-read speech (Earnings-21, Yodas), where their unbiased baselines often trail the standard Whisper model.
• Sensitivity to Distractors: Speech LLMs are highly sensitive to the number of distractors. Adding 250 distractors causes BWER to degrade dramatically (39% to 570% relative increase). Furthermore, their performance is sensitive to the order of words in the prompt, performing better when target words appear earlier.
• Filtering Impact: Applying a filtering stage using an auxiliary model (Qwen3-Omni) to prune the bias list recovers some performance loss by reducing the prompt to a few likely-relevant words. However, even with filtering, speech LLMs still trail behind Method B on non-read test sets.

Significance and Claims
The paper concludes that the choice of method depends on the specific application constraints:

• Dedicated Context Biasing: Remains the more robust choice when a clean word list is available and resilience to noisy or long lists is required. These methods offer a dedicated, controllable interface that maintains performance regardless of distractor count.
• Speech LLMs: Offer a more flexible interface capable of handling free-form descriptions and semantic context (e.g., disambiguating "forest" vs. "Far East" based on geographic cues). However, this flexibility comes at the cost of prompt sensitivity and the practical necessity of an additional filtering stage to handle large lists of potential words.

The authors emphasize that while speech LLMs show promise, their current limitations in generalizing to non-read speech and their sensitivity to prompt composition make dedicated biasing methods superior for scenarios requiring high reliability with specific word lists.