Contextual Earnings-22: A Speech Recognition Benchmark with Custom Vocabulary in the Wild

This paper introduces Contextual Earnings-22, a new open benchmark dataset featuring realistic custom vocabulary contexts to address the plateau of speech-to-text accuracy on academic tests, demonstrating that both keyword prompting and keyword boosting approaches achieve significant improvements when scaled to large-scale systems.

The Gist

The Big Problem: The "Perfect" Translator Who Misses the Point

Imagine you have a super-smart translator who can speak every language fluently. On standard tests (like translating a news article or a casual chat), this translator gets a 99% score. They are perfect at general words like "the," "run," "happy," and "cat."

But, you hire this translator for a very specific job: transcribing a high-stakes business meeting about a new tech company called "Nebula" and a CEO named "Dr. Aris Thorne."

Even though the translator is 99% accurate on general words, they keep making a fatal mistake: they write "Nebula" as "Nebula" (correct) but then accidentally change "Dr. Aris Thorne" to "Dr. Arthur Thorne" or "Dr. A. Thorne."

In the real world, getting the general grammar right doesn't matter if you get the names wrong. The transcript is useless. This is the problem the paper addresses: Speech recognition is great at general words, but terrible at the specific, custom words that actually matter in real life.

The Solution: A New "Driver's License Test" for AI

The researchers realized that current tests for AI speech systems are like a driving test on an empty, straight highway. The AI passes easily. But in the real world, you have to navigate a busy city with specific street names, construction zones, and confusing signs.

To fix this, they built a new benchmark called Contextual Earnings-22.

Think of this benchmark as a specialized driving test designed specifically for navigating the "city" of corporate earnings calls.

• The Course: They took real recordings of financial meetings (where people talk about specific companies, products, and people).
• The Challenge: They chopped these long meetings into short 15-second clips.
• The "Context" (The Clue): For each clip, they gave the AI a "cheat sheet" (a list of names and terms) that might be mentioned.
  • Level 1 (Local Context): The cheat sheet only has the exact names spoken in that 15-second clip. (Easy mode: "Here is the name, please say it correctly.")
  • Level 2 (Global Context): The cheat sheet has the names from the entire hour-long meeting, including names that weren't spoken in this specific 15-second clip. (Hard mode: "Here is a list of 50 names. Only 3 are in this clip. Don't get confused by the other 47.")

How They Tested the AI

They took six different "drivers" (AI speech systems) and put them through this test. Some drivers use Keyword Prompting (telling the AI: "Hey, try to listen for these words"), and others use Keyword Boosting (mathematically nudging the AI to prioritize these words).

They measured two things:

1. The "Overall Score" (WER): How many total words were wrong?
2. The "Name Score" (Keyword F-Score): Did they get the specific, important names right?

What They Found

1. The "Name Score" matters more than the "Overall Score."
When they gave the AI the cheat sheet (context), the "Name Score" went up dramatically. The AI stopped confusing "Exane" with "Examine" or "Dan" with "Don."

• Analogy: It's like giving a chef a list of ingredients. Without the list, they might guess "salt" instead of "saffron." With the list, they get the expensive, specific ingredient right, even if they still mess up a few other minor words.

2. The "Distractor" Problem.
When the AI was given the "Global Context" (a long list with names that weren't in the clip), some AI systems got confused. They started hallucinating, inserting names from the list that were never spoken.

• Analogy: Imagine a security guard with a list of 100 VIPs. If a regular person walks by, the guard might mistakenly think, "Oh, that looks like VIP #45!" and stop them. The AI did the same thing, inserting names just because they were on the list.

3. Different AI systems handle the "Cheat Sheet" differently.
Some systems were great at using the list to get names right but started making weird mistakes elsewhere (like changing the language or repeating words). Others were more stable but didn't improve the names as much.

The Takeaway

The paper concludes that we need to stop judging speech AI only by how well it handles general conversation. We need to test it on specific, real-world scenarios where getting the custom vocabulary right is the difference between a useful transcript and a useless one.

They have released this new dataset and the testing tools to the public, hoping that other researchers will use this "specialized driving test" to build AI that doesn't just sound good, but actually gets the important details right.

In short: They built a better test to stop AI from being a "polite but clueless" listener and turn it into a "sharp, detail-oriented" assistant.

Technical Summary

1. Problem Statement

Despite high accuracy on standard academic benchmarks, Speech-to-Text (STT) systems often fail in high-stakes industrial applications (e.g., financial earnings calls). The core issue is a mismatch between benchmark metrics and real-world utility:

• The Saturation Gap: Standard Word Error Rate (WER) improvements have plateaued on general vocabulary. However, real-world usability is disproportionately determined by the accurate recognition of rare, context-defined custom vocabulary (e.g., specific company names, product lines, and executive names).
• Lack of Standardization: Existing research on "contextual biasing" (improving recognition of specific terms) relies on fragmented, synthetic, or private datasets. There is no standardized public benchmark that pairs realistic custom vocabulary with both idealized (precise) and deployment-realistic (noisy/distractor-heavy) evaluation regimes.

2. Methodology: The Contextual Earnings-22 Dataset

The authors introduce Contextual Earnings-22, a new open dataset built upon the Earnings-22 corpus, designed to bridge the gap between academic research and industrial deployment.

Data Construction Pipeline

1. Keyword Extraction: Using an LLM (GPT-5), the authors extract candidate keywords from transcripts, focusing on three categories: Person, Company, and Product names.
2. Normalization: A deterministic post-processing step ensures stability by de-duplicating surface forms, normalizing punctuation/whitespace, and filtering generic strings.
3. Segmentation & Alignment:
   • The system locates keyword mentions in the transcript.
   • It extracts 15-second audio clips centered on these mentions using forced alignment (wav2vec-based).
   • Each clip is paired with a local context (keywords in the clip) and a global context (the full list of keywords from the entire 1-hour call).
4. Manual Review & Correction: A critical step where human reviewers corrected artifacts (e.g., misheard names, casing errors, <unk> tags).
   • Result: 98.7% of samples are free of inaudible tags, and 29.5% of clips received word-level corrections.

Evaluation Regimes

The benchmark evaluates systems under two distinct conditions to mirror real-world scenarios:

• Local Context: The system receives only the keywords present in the specific 15-second clip. This tests the system's ability to leverage precise, relevant context.
• Global Context: The system receives the entire call's keyword inventory (including keywords not spoken in the clip). This introduces distractors, testing the system's robustness against false positives when users provide long, noisy custom vocabularies.

3. Key Contributions

• Standardized Benchmark: The first public dataset specifically designed to evaluate contextual STT with realistic custom vocabulary, offering both local and global context regimes.
• Comprehensive Baselines: The authors established six strong baselines covering the two dominant approaches to contextualization:
  1. Keyword Prompting: Using textual prompts (via APIs like Deepgram, OpenAI Whisper, AssemblyAI, and Whisper OSS).
  2. Keyword Boosting: Using decoding-side integration (via CTC-WS pipelines, including Argmax's Parakeet-v2).
• Dual-Metric Evaluation: Moving beyond standard WER, the paper emphasizes Keyword-Centric Metrics (Precision, Recall, F-score) to measure the specific utility of custom vocabulary recognition.
• Open Source Release: The dataset includes audio clips, corrected transcripts, context lists, and an open-source evaluation harness to ensure reproducibility.

4. Results & Findings

The paper evaluated six STT systems across no-context, local-context, and global-context settings.

• Context Improves Keyword Recognition: All systems showed significant improvements in Keyword F-score when context was provided, confirming that contextual conditioning effectively targets the "long tail" of rare vocabulary.
• WER vs. Keyword Accuracy Decoupling:
  • Improvements in keyword accuracy did not always correlate with lower overall WER.
  • Some systems (e.g., OpenAI Whisper in this setup) showed increased WER under context despite better keyword F-scores, indicating that context injection can introduce side effects (hallucinations or artifacts).
• Local vs. Global Trade-offs:
  • Local Context: Generally yielded higher F-scores (better precision/recall) as the risk of inserting non-spoken words was minimized.
  • Global Context: Stressed precision. Systems often suffered from false positives (hallucinating keywords from the distractor list), leading to lower F-scores.
• System Differences:
  • Prompting vs. Boosting: Both approaches reached comparable high accuracy when scaled, but their failure modes differed.
  • Artifacts: Prompt-based systems exhibited specific artifacts, including hallucinations (inserting words not spoken), partial outputs, and occasional language switching.

5. Significance

• Redefining Success: The paper argues that Keyword F-score is a more critical metric than aggregate WER for high-stakes domains where specific entities determine transcript utility.
• Realistic Stress Testing: By introducing the "Global Context" regime with distractors, the benchmark exposes a critical weakness in current systems: the inability to distinguish between relevant and irrelevant custom vocabulary in real-world deployment scenarios.
• Accelerating Research: By providing a standardized, high-quality dataset with reproducible baselines, Contextual Earnings-22 enables fair "apples-to-apples" comparisons between prompting and boosting strategies, fostering further progress in making STT robust for specialized industries.

In conclusion, Contextual Earnings-22 demonstrates that while STT systems have mastered general vocabulary, the frontier of progress lies in contextual conditioning. The benchmark reveals that while context improves specific term recognition, it introduces new challenges regarding precision and artifact generation that must be addressed for reliable industrial adoption.