Technical Summary: WorldSpeech: A Multilingual Speech Corpus from Around the World

Problem Statement

Automatic Speech Recognition (ASR) systems currently perform robustly on high-resource languages with abundant paired audio-transcript data. However, accuracy degrades sharply for the majority of the world's languages due to a scarcity of publicly available, human-aligned data. Existing solutions face significant limitations: crowd-sourced corpora like Common Voice offer broad coverage but limited hours per language; audiobook collections provide volume but cover few languages; and large-scale datasets like MOSEL and YODAS rely on pseudo-labels generated by ASR rather than human ground truth. Furthermore, existing approaches leveraging parliamentary records (e.g., VoxPopuli, EuroSpeech) are largely restricted to Europe due to data access restrictions in other regions and the failure of alignment pipelines when initial ASR models perform poorly on low-resource languages, preventing the recovery of ground-truth transcripts.

Methodology

The authors introduce WORLDSPEECH, a multilingual speech corpus constructed by extending the parliamentary alignment recipe to international broadcasters, national public broadcasters, and public-domain audiobook archives.

Data Collection and Standardization

The corpus aggregates data from 79 distinct sources across 82 countries, including parliamentary proceedings, international broadcasts (e.g., VOA, RFA, RFE/RL), and public-domain audiobooks (e.g., LibriVox, Aozora). The pipeline involves:

• Format Standardization: Converting diverse audio formats (MP4, MP3, HLS streams) to mono 24 kHz and transcripts (HTML, XML, DOCX, PDF, SRT) to plain text.
• Robust Text Extraction: Handling complex PDF layouts (e.g., two-column Hansard documents) via image rendering and column cropping. For scripts where standard extraction fails (e.g., Nastaliq Urdu, Burmese), the pipeline employs OCR fallbacks using Tesseract and the Surya neural OCR engine.
• Normalization: Applying language-specific cleaning, such as stripping Arabic harakat, normalizing tone marks for Cantonese, and handling code-switching in multilingual parliaments via language detection.

Alignment Pipeline

The core of the methodology is a two-stage coarse-to-fine alignment strategy adapted from EuroSpeech:

1. Segmentation: Long-form audio is segmented into 3–30 second utterances using Silero VAD.
2. ASR Selection: An ablation study selects the optimal ASR model (e.g., Whisper-large-v3-turbo, MMS-1B with adapters, or community fine-tunes) for each language based on which maximizes the fraction of segments passing a Character Error Rate (CER) < 0.3 threshold on a 10-hour sample.
3. CER-based Matching: The system computes the CER between ASR hypotheses and candidate spans in the human transcript. Segments are retained only if the minimum CER falls below 0.3. The ASR output is used solely for search, not as ground truth.

Iterative Alignment Refinement

To address the "cold start" problem where poor initial ASR performance prevents alignment, the authors propose an iterative refinement scheme:

1. Pass 1: Align audio with transcripts using the initial ASR model.
2. Fine-tuning: Fine-tune the ASR model on the segments successfully aligned in Pass 1.
3. Pass 2: Re-align the remaining unaligned audio using the fine-tuned model.
   This process recovers segments that were initially mismatched due to hallucinations or script errors, increasing aligned hours without new data collection. The authors found that a third pass yields diminishing returns (+0.2% to +8.8%) compared to the second pass.

Key Contributions

1. WORLDSPEECH Corpus: A publicly available corpus containing 64,970 hours of aligned audio-transcript data across 76 languages. It is the first public corpus to combine broad coverage with significant depth, featuring:
   • 24 languages with >1,000 hours.
   • 28 languages with >500 hours.
   • 37 languages with >200 hours.
   • Coverage of regional and dialectal variants (e.g., Québécois French, Austrian German, Bahraini Arabic) often absent from existing datasets.
2. Iterative Refinement Technique: A method to increase aligned data volume by 19.5% to 201.1% per language by iteratively fine-tuning the alignment ASR, specifically benefiting languages with weak initial ASR performance (e.g., Burmese, Lao, Khmer).
3. Performance Gains: Demonstration that fine-tuning existing multilingual ASR backbones on WORLDSPEECH significantly reduces Word Error Rate (WER) across typologically diverse languages.

Results

The authors evaluated the corpus by fine-tuning Whisper-large-v3-turbo on the WORLDSPEECH training split and testing on public benchmarks (FLEURS) or held-out test sets.

• WER Reduction: Fine-tuning resulted in an average relative WER reduction of 63.5% across 11 typologically diverse languages.
• Low-Resource Impact: The most significant gains were observed in languages where the zero-shot baseline WER exceeded 1.0 (indicating more errors than reference words). For example:
  • Samoan: WER reduced from 4.72 to 0.39 (−91.7%).
  • Lao: WER reduced from 2.47 to 0.76 (−69.4%).
  • Romansh: WER reduced from 1.31 to 0.17 (−87.5%).
• Data Scaling: An ablation study showed that WER decreases monotonically with available hours, with the steepest improvements occurring in the first 200 hours and diminishing returns beyond 500 hours.

Significance

The paper positions WORLDSPEECH as a critical resource for closing the gap between current publicly available aligned data and the volume required to train high-accuracy ASR models for low-resource languages. By providing the first publicly available corpus that combines per-language depth (with 24 languages exceeding 1,000 hours of human-labeled data) with broad global coverage (76 languages), WORLDSPEECH enables the training and evaluation of ASR systems for previously underserved languages and dialects. The iterative refinement strategy further offers a scalable method to maximize the utility of existing public archives without requiring additional data collection efforts.