💬 NLP

Whisper: Courtside Edition Enhancing ASR Performance Through LLM-Driven Context Generation

The paper introduces "Whisper: Courtside Edition," a novel multi-agent LLM pipeline that significantly improves domain-specific ASR performance on NBA commentary by generating context-aware prompts to guide the decoder, achieving a 17.0% relative reduction in word error rate without requiring model retraining.

Original authors: Yonathan Ron, Shiri Gilboa, Tammuz Dubnov

Published 2026-02-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yonathan Ron, Shiri Gilboa, Tammuz Dubnov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, world-class translator named Whisper. This translator has read almost every book ever written and can understand languages from every corner of the globe. If you ask Whisper to translate a normal conversation about the weather or dinner plans, it does a perfect job.

But, if you ask Whisper to translate a live NBA basketball game, it starts to stumble.

Why? Because the game is full of fast-paced slang, weird player names (like "Giannis Antetokounmpo"), and specific terms ("pick and roll," "alley-oop"). Whisper hears "Giannis" and thinks, "Hmm, that sounds like 'Yanis'." It hears "pick and roll" and thinks, "Oh, they must mean a 'picker roll'." It's like a translator who knows English perfectly but has never stepped foot on a basketball court.

This paper introduces a solution called "Whisper: Courtside Edition." It doesn't try to retrain Whisper (which would be like sending the translator back to school for five years). Instead, it gives Whisper a smart assistant right before it starts translating.

Here is how it works, using a simple analogy:

The Problem: The Translator in the Dark

Imagine Whisper is a chef trying to cook a complex dish (the basketball game) but they are working in a dark kitchen. They can hear the ingredients being chopped (the audio), but they can't see the recipe. So, they guess.

They hear a name that sounds like "Christmas" and write down "Christmas Por Zingas" instead of "Kristaps Porzingis."
They hear "fast break" and write "fast brake."

The Solution: The "Courtside" Assistant

The authors built a multi-agent pipeline. Think of this as a team of three specialized experts sitting right next to the chef (Whisper) before the cooking starts.

The Scout (Topic Agent): This agent listens to the audio for a split second and whispers to the chef: "Hey, we are cooking an NBA game, not a cooking show. Get ready for sports terms!"
The Statistician (Name Agent): This agent has a giant list of every NBA player's name. If the chef hears a sound that might be a player, the Statistician checks the list and says, "That sounds like 'Stephen Curry,' not 'Stephen Curry's brother.' Make sure you spell it right."
The Coach (Jargon Agent): This agent knows the slang. If the chef hears "pick and roll," the Coach taps the chef on the shoulder and says, "Don't write 'picker roll.' The correct term is 'pick and roll'."

The Magic Trick: The "Prompt"

Instead of letting the chef cook and then trying to fix the mistakes afterward (which is hard because the chef can't hear the audio again), this team gives the chef a cheat sheet before they start.

In technical terms, this is called an "initial prompt."

The team takes the raw list of names and terms, turns it into a short, natural sentence (e.g., "This is an NBA game featuring Stephen Curry and the pick-and-roll play"), and feeds it to Whisper.
Whisper reads this sentence, gets "primed" with the right context, and then listens to the audio again.
Because Whisper now knows what to expect, it stops guessing and starts recognizing. It's like the chef suddenly turning on the lights in the kitchen; they can finally see the ingredients clearly.

The Results: A Winning Game

The team tested this on 421 real NBA commentary clips.

Before the assistant: Whisper made mistakes about 21.7% of the time (Word Error Rate).
After the assistant: Mistakes dropped to 18.0%.
The Win: This is a 17% improvement, which is huge in the world of AI.
Safety: The system is careful. It only suggests changes it is sure about. If it's not sure, it leaves the original text alone. This means it fixes errors without accidentally creating new ones.

Why This Matters

Usually, to fix a specific problem like sports commentary, you have to spend thousands of dollars and months of time to retrain the AI model from scratch.

This paper shows a smarter, cheaper way: Don't retrain the brain; just give it better context.

It's like realizing you don't need to teach your translator how to play basketball; you just need to hand them a rulebook and a roster list before the game starts. This approach can be used for any specialized field—doctors, lawyers, or engineers—without needing to rebuild the AI every time.

In short: They built a "smart co-pilot" that whispers the right words to the AI before it speaks, turning a confused translator into a courtside expert.

1. Problem Statement

Automatic Speech Recognition (ASR) systems, even state-of-the-art models like OpenAI's Whisper, struggle with domain-specific speech. While they perform well on general-domain audio, they exhibit systematic failures in specialized fields (e.g., sports commentary, medical dictation) due to:

Proper Noun Errors: Mishearing player names (e.g., "Giannis Antetokounmpo" $\rightarrow$ "yanis anteto kumbo").
Jargon Corruption: Incorrectly transcribing technical terms (e.g., "pick and roll" $\rightarrow$ "picker roll").
Contextual Misinterpretation: Failing to resolve homophones or rapid speech patterns unique to the domain.

Traditional solutions like acoustic model fine-tuning or language model adaptation require expensive retraining and large annotated corpora. The authors propose a solution that enhances ASR performance without retraining, leveraging the model's existing capabilities through intelligent prompting.

2. Methodology: Multi-Agent LLM Pipeline

The core innovation is a multi-agent Large Language Model (LLM) pipeline that acts as a "context generator" for Whisper. Instead of post-processing the final transcript, the system intercepts Whisper's initial transcript to generate a specialized initial_prompt for a second decoding pass.

The Pipeline Architecture (P4 - Full Multi-Agent)

The system employs six specialized agents to process the first-pass transcript and construct a compact, natural-language prompt:

Topic Classification Agent: Identifies the domain context (e.g., "NBA basketball commentary") to activate relevant knowledge bases.
Named Entity Recognition (NER) Agent: Extracts person names and performs fuzzy matching (using Levenshtein distance and Jaro–Winkler similarity) against official NBA rosters to correct spellings.
Jargon Extraction Agent: Identifies domain-specific terminology using keyword extraction (RAKE/YAKE) and cross-references a basketball glossary.
Decision Filtering Logic: Critical validation agents that prevent over-correction. They verify that proposed names are actual corrections, ensure jargon adds semantic value, and enforce prompt length constraints (Whisper's limit is 224 tokens, with the last tokens having the most influence).
Candidate Selector: Resolves conflicts between sources and prunes the list to a high-confidence shortlist.
Sentence Builder: Assembles the topic, corrected names, and jargon into a single concise natural-language sentence. This format is crucial because Whisper's decoder is optimized for natural text continuation, and placing high-value tokens near the end of the prompt maximizes their influence.

Mechanism

The pipeline exploits Whisper's initial_prompt parameter. By injecting a context-rich sentence (e.g., "NBA game commentary featuring Stephen Curry, Giannis Antetokounmpo, and pick-and-roll plays") before the audio decoding begins, the model biases its probability distribution toward the correct domain vocabulary while still reconciling the text with the acoustic evidence.

3. Key Contributions

Novel Multi-Agent Architecture: The first application of a specialized multi-agent LLM system to enhance speech recognition via Whisper's prompting mechanism, decomposing tasks into topic detection, entity normalization, and jargon handling.
Prompt-Based Domain Adaptation: Demonstrates that domain adaptation can be achieved without costly model retraining, relying instead on dynamic, intelligent prompt generation.
Comprehensive Evaluation: A rigorous statistical evaluation on a real-world dataset of 421 NBA commentary segments, including a detailed error taxonomy and failure mode analysis.
Generalizable Framework: A system design that requires only domain knowledge bases (rosters, glossaries) and LLM access, making it applicable to other domains (medical, legal) without architectural changes.

4. Experimental Results

The system was evaluated on 421 NBA commentary segments (median length 15s) using Word Error Rate (WER) as the primary metric.

Baseline Performance: Whisper (medium.en) achieved a WER of 0.217.
Best Performance (P4 - Full Multi-Agent):
- WER: Reduced to 0.180.
- Improvement: A 17.0% relative reduction in WER (statistically significant, $p < 0.001$ ).
- Segment Analysis: 40.1% of segments showed improvement, while only 7.1% showed degradation.
Comparison with Variants:
- P1 (Topic-Only): Increased WER (0.238), proving that broad context without specific vocabulary can mislead the model.
- P2 (LLM Post-Correction): Minimal impact (WER 0.217), confirming that post-processing cannot fix misheard audio without access to the signal.
- P3 (Names-Enhanced): Moderate improvement but higher degradation rates (20.7%) due to lack of filtering, highlighting the necessity of the decision-filtering agents in P4.

Key Success Cases:

Proper Nouns: Corrected "anteto kumbo" $\rightarrow$ "Antetokounmpo".
Jargon: Corrected "fast brake" $\rightarrow$ "fast break".
Disambiguation: Resolved "the pass to 80" $\rightarrow$ "the pass to A.D." (Anthony Davis).

5. Significance and Implications

Scalability: This approach offers a practical, scalable alternative to fine-tuning, which is often cost-prohibitive for niche domains.
Model Agnosticism: The method does not lock users into a specific ASR vendor; it enhances the output of existing models using external knowledge.
Real-World Applicability: The pipeline adds a median latency of 3.3 seconds per segment with low API costs (~$0.008/segment), making it viable for near-real-time applications like live sports captioning, specialized transcription services, and domain-specific voice assistants.
Future Directions: The authors suggest extending this framework to medical and legal domains, optimizing for real-time streaming via agent parallelization, and integrating multimodal context (e.g., video analysis) to further reduce ambiguity.

In conclusion, "Whisper: Courtside Edition" proves that intelligent prompting driven by multi-agent LLMs can significantly bridge the gap between general-purpose ASR and domain-specific accuracy, offering a robust, non-invasive path to specialized speech recognition.