HALAS: A Human-Annotated Dataset of Hallucinations of Modern ASR Systems

Technical Summary: HALAS – A Human-Annotated Dataset of Hallucinations of Modern ASR Systems

Problem Statement
End-to-end Automatic Speech Recognition (ASR) systems are prone to generating "hallucinations"—erroneous text in transcriptions that does not correspond to the spoken audio. While existing mitigation and detection methods are often evaluated on non-speech audio or artificially corrupted datasets, there is a lack of rigorous benchmarks based on naturally occurring hallucinations in real-world speech. Current detection approaches, including those relying on reference-based validation or non-reference proxy metrics, have not been sufficiently tested against human-annotated hallucinations in unprocessed speech, limiting their practical applicability.

Methodology
The authors introduce HALAS (Hallucination Annotations for Large-scale ASR Systems), the first human-annotated dataset of naturally occurring hallucinations. The dataset creation process involved the following steps:

1. Data Source: The authors utilized the Earnings 22 (E22) dataset, comprising 119 hours of English earnings call recordings from 27 countries.
2. Model Selection: Seven state-of-the-art ASR models were selected based on their performance on the Open ASR Leaderboard: four versions of OpenAI's Whisper (large v3, large v3 Turbo, large v2, and a fine-tuned Crisper Whisper) and three Nvidia Nemo models (Canary-1B, Canary-1B-Flash, and Parakeet-TDT v2).
3. Candidate Selection: To maximize the prevalence of hallucinations, the authors used inter-model Word Error Rate (WER) as a proxy for audio difficulty. Audio segments where models produced significantly different transcriptions were prioritized. Preprocessing involved removing files marked as "inaudible" or "foreign language," and normalizing predictions to remove repetitive looping for metric calculation purposes.
4. Human Annotation: Ten professional annotators (English B2+) labeled the data. Crucially, annotations were performed by comparing ASR predictions directly against the audio recording, not the reference transcripts provided by the E22 dataset (which were frequently found to be incorrect). Annotators marked specific spans as "Hallucination," "Looping," or "Looping Hallucination." A third annotator acted as an arbitrator to resolve disagreements, resulting in a high initial agreement (Cohen's kappa = 0.87).
5. Dataset Structure: The final dataset contains 3,611 audio files with span-level labels for all seven models. It is split into training (2,866 files, 33.6% hallucination rate) and test (745 files, 22.6% hallucination rate) sets, stratified by source meeting, WER, and duration.

Key Contributions
The paper presents three primary contributions:

1. The HALAS Dataset: A novel, publicly available dataset of human-labeled hallucinations from seven SOTA ASR models on real speech, offering span-level granularity.
2. Empirical Analysis: The first comprehensive analysis of hallucination patterns in SOTA ASR models on real data, revealing strong cross-model vocabulary overlap and the occurrence of hallucinations even in low-WER scenarios.
3. Benchmarking Framework: A rigorous benchmark for evaluating hallucination detection methods against naturally occurring, human-annotated errors, moving beyond artificial or non-speech evaluations.

Results and Analysis

• Hallucination Patterns: All seven models exhibited hallucinations, with rates ranging from 21.4% to 43.8%. The analysis revealed that hallucinations are heavily skewed toward a small set of frequent phrases (e.g., "you," "thank you," "okay"). On average, 55% of marked hallucinations corresponded to the top 10 phrases for a given model.
• Cross-Model Similarity: Jensen-Shannon similarity scores indicated strong distributional overlap in hallucinated phrases across models, particularly among the Whisper variants.
• Severity Assessment: Using an LLM (GPT-4o mini) to classify severity, the authors found that hallucinations are often binary: either benign fillers (minor) or major semantic disruptions (severe), with "moderate" errors being less frequent. Models like Parakeet and Crisper Whisper tended toward minor insertions, while Canary and Whisper v2 exhibited higher rates of severe errors.
• Detection Performance:
  • Proxy Metrics: Standard metrics like WER and Perplexity showed limited efficacy (ROC-AUC ~0.60–0.72) because hallucinations in HALAS are often fluent and concise. Semantic metrics (BERTScore, SeMaScore) and Character Error Rate (CER) performed better, reaching an ROC-AUC of approximately 0.81.
  • SOTA Methods: When benchmarking detection methods, reference-free decoder embedding (DE) classifiers outperformed LLM-based reference methods. The best-performing detector (a multi-layer DE classifier using layers 2, 13, and 23) achieved an F1 score of 56.1% on the HALAS test set. In contrast, LLMs (GPT-4o mini, Gemini 2.0 Flash) achieved F1 scores around 41%, often suffering from low precision or low recall.
  • Generalization: Detectors trained on HALAS demonstrated the ability to generalize across different ASR architectures (OWN vs. OTHER training regimes) and performed even better on out-of-domain non-speech data, suggesting HALAS captures fundamental error patterns.

Significance
The paper establishes HALAS as the first rigorous, non-artificial benchmark for ASR hallucination detection and mitigation. The authors argue that the problem of detecting hallucinations in natural speech is significantly more challenging than previously demonstrated on artificial datasets, as evidenced by the modest performance of current state-of-the-art detection methods (F1 ~53–56%). The dataset enables the development of detectors that can identify hallucinations without relying on ground-truth reference text, a critical capability for real-world deployment where reference transcripts are unavailable.