BlasBench: An Open Benchmark for Irish Speech Recognition

1. Problem Statement

The paper identifies a critical gap in the field of Automatic Speech Recognition (ASR) for the Irish language (Gaeilge): the lack of a standardized, open benchmark that allows for fair comparison of end-user ASR systems.

• Incomparable Results: Existing efforts (e.g., Trinity College Dublin's ABAIR group, community fine-tunes on HuggingFace, and multilingual benchmarks like FLEURS or ML-SUPERB) cannot be reliably compared. They use different datasets, different versions of Common Voice, and, crucially, different text normalization protocols.
• Loss of Linguistic Information: Standard, language-agnostic normalization (lowercasing, stripping punctuation) destroys linguistically significant features of Irish, such as:
  • Fadas: Acute accents (á, é, í, ó, ú) which are phonemically contrastive (e.g., fear "man" vs. féar "grass").
  • Initial Mutations: Grammatical markers like lenition (bean → bhean) and eclipsis (bean → mbean).
• Evaluation Bias: Single-dataset evaluation (specifically on Common Voice) fails to reveal generalization gaps, leading to overestimated performance metrics that do not hold up on diverse, real-world data.

2. Methodology: The BlasBench Framework

The authors introduce BlasBench, an open evaluation harness designed to standardize Irish ASR evaluation.

A. Evaluation Pipeline

The pipeline consists of four distinct stages:

1. Inference: Audio (16kHz) is processed by a model wrapper (standardized API) to produce raw text.
2. Irish-Aware Normalization: Both the reference transcription and the model hypothesis are processed through a custom normalizer.
   • Preservation: It guarantees the survival of fadas (using NFC normalization) and preserves initial mutations (lenition/eclipsis).
   • Innovation: This is the first standalone, open-source normalizer specifically designed for Irish ASR scoring.
3. Scoring: Word Error Rate (WER) and Character Error Rate (CER) are calculated via global aggregation (summing all substitutions, insertions, and deletions across the entire dataset) rather than per-sentence averaging, to avoid statistical bias.
4. Statistical Confidence: Bootstrap 95% Confidence Intervals (CIs) are computed using 1,000 resamples to ensure result stability.

B. Datasets

To measure both in-distribution and out-of-distribution performance, BlasBench utilizes two distinct datasets:

• Common Voice ga-IE (v25.0): 874 community-recorded read-speech utterances.
• FLEURS ga-IE: 842 professionally recorded native-speaker utterances.
• Significance: Using both datasets exposes generalization failures that single-dataset evaluations miss.

C. Systems Evaluated

The benchmark tested 12 systems across four architecture families:

1. Whisper Variants: Medium, Large-v2, Large-v3, and Large-v3-turbo (evaluated zero-shot as Irish is absent or unreported in their training data).
2. wav2vec2 CTC: Community fine-tunes of XLS-R and XLSR-53 on Common Voice.
3. Multilingual Foundation Models: Meta's MMS-1B-All and the new omniASR (LLaMA-style decoders with wav2vec2 encoders) in 300M and 7B parameter sizes.
4. Commercial: Microsoft Azure Speech Services.

3. Key Contributions

1. BlasBench Repository: An open-source evaluation harness (GitHub) that includes the Irish-aware normalizer, scoring scripts, and a standardized API for adding new models.
2. First Comparative Benchmark: The first study to compare end-user ASR systems across different architecture families under a shared, Irish-specific protocol.
3. Public Artifacts: Release of per-utterance predictions, aggregate results, and run metadata for all 12 systems, allowing future researchers to compare new models without re-running baselines.
4. Irish-Aware Normalizer: A standalone tool that correctly handles Irish orthography, addressing a specific linguistic bottleneck in low-resource language evaluation.

4. Key Results

The evaluation yielded several surprising and critical findings:

• Catastrophic Failure of Whisper: All Whisper variants evaluated performed worse than random guessing, with WER exceeding 100% on both datasets.
  • The models emitted fluent English unrelated to the input audio.
  • Performance degraded monotonically from v2 to v3 to turbo (e.g., v3-turbo reached 587.6% WER on FLEURS).
• Generalization Gap: Models fine-tuned specifically on Common Voice showed massive performance drops when tested on FLEURS.
  • Common Voice-trained models degraded by 33–43 WER points on FLEURS.
  • In contrast, massively multilingual pre-trained models (omniASR, MMS) showed much smaller gaps (7–10 points), indicating better generalization.
• Best Performers:
  • Commercial: Microsoft Azure achieved 22.2% WER on Common Voice but dropped to 57.5% on FLEURS.
  • Open Source: The omniASR LLM 7B was the best open model, achieving 30.65% WER on Common Voice and 39.09% WER on FLEURS.
  • wav2vec2: Community fine-tunes of wav2vec2 (XLS-R) were the only viable open architecture family, though they still lagged behind commercial solutions.
• Open vs. Closed Gap: While ABAIR's closed system (Fotheidil) reports 10.9% WER, this is on private data with a different normalizer. Under BlasBench's protocol, the gap between the best open model (omniASR 7B) and the best reported closed system is narrower than headline numbers suggest, but data scarcity remains the primary bottleneck.

5. Significance and Conclusion

• Reproducibility: BlasBench establishes a stable, deterministic scoring function essential for automated research pipelines and credible comparison of ASR models.
• Linguistic Integrity: By preserving fadas and mutations, the benchmark ensures that error rates reflect true linguistic understanding rather than normalization artifacts.
• Data Bottleneck: The paper concludes that while architecture matters (wav2vec2 is currently superior to Whisper for Irish), the primary constraint on progress is the lack of labeled Irish audio data.
• Future Direction: The authors emphasize that single-dataset evaluation is insufficient; future benchmarks for low-resource languages must include cross-corpus analysis to detect generalization failures.

In summary, BlasBench provides the first rigorous, open framework for Irish ASR, revealing that current "state-of-the-art" multilingual models (like Whisper) are ineffective for Irish, while highlighting the urgent need for more high-quality, dialect-balanced training data.