RO-N3WS: Enhancing Generalization in Low-Resource ASR with Diverse Romanian Speech Benchmarks

1. Problem Statement

Automatic Speech Recognition (ASR) systems have achieved significant success in high-resource languages like English, but they struggle with low-resource languages such as Romanian. Existing Romanian datasets (e.g., Common Voice, VoxPopuli, FLEURS) suffer from several limitations:

• Domain Constraints: They are often restricted to read speech, parliamentary recordings, or isolated words, lacking the variability of real-world spontaneous speech.
• Lack of Expressiveness: They fail to capture prosodic variation, emotional delivery, and conversational nuances found in films, audiobooks, or podcasts.
• Generalization Gap: Pre-trained multilingual models (like Whisper and Wav2Vec 2.0) show strong zero-shot performance but degrade significantly when faced with domain shifts (e.g., moving from news broadcasts to film dialogue) or out-of-distribution (OOD) conditions.

There is a critical need for a benchmark dataset that includes both in-domain (broadcast news) and out-of-distribution (expressive, spontaneous) speech to rigorously evaluate and improve ASR generalization in low-resource settings.

2. Methodology

Dataset Construction (RO-N3WS)

The authors introduce RO-N3WS, a curated Romanian speech dataset comprising 126 hours of transcribed audio. The construction pipeline involved:

• Sources:
  • In-Domain (105 hours): Broadcast news from two major Romanian channels (ProTV and Antena 1). This includes studio and field reports.
  • Out-of-Distribution (21 hours): Diverse sources including audiobooks, Romanian films, children's stories, and conversational podcasts.
• Preprocessing:
  • Audio was scraped, converted to WAV, and initially transcribed using Whisper-timestamped to obtain word-level timestamps.
  • Cleaning: Segments with overlapping speech, music, foreign languages, or poor segmentation were discarded (approx. 60% of Antena 1 data was removed).
  • Annotation: A team of 15 annotators manually corrected transcripts, restoring diacritics, expanding numbers (e.g., "100" to "o sută"), and correcting spelling while preserving phonetic fidelity.
• Splits: The data is stratified into Training (85%), Validation (10%), and Test (5%) sets, ensuring that segments from the same long-form video remain in the same fold to prevent data leakage.

Dataset Analysis

The authors performed quantitative analyses to demonstrate the dataset's richness compared to existing benchmarks:

• Named Entity Density: RO-N3WS exhibits the highest density of named entities (11.35 per 100 tokens), surpassing Echo, FLEURS, and VoxPopuli. This is crucial for testing ASR handling of proper nouns, dates, and quantities.
• Prosodic Variation: Using Parselmouth (Praat), the authors measured pitch and intensity. The OOD subset showed the highest pitch standard deviation (71.92 Hz), indicating high emotional expressiveness, while the in-domain news subset showed the highest mean intensity (63.65 dB), reflecting professional broadcast projection.

Experimental Setup

• Models Evaluated:
  • Open-Source: Wav2Vec 2.0 (VoxPopuli fine-tuned), Whisper Small, Whisper Large, and Whisper Small fine-tuned on the Echo dataset.
  • Commercial (Zero-shot only): Vatis, Microsoft Transcribe, and Google Chirp (USM).
• Evaluation Metrics: Word Error Rate (WER) was the primary metric. To handle formatting discrepancies (e.g., numbers written as digits vs. words), the authors used relaxed reference annotations that account for semantic equivalence.
• Scenarios:
  1. Zero-Shot: Evaluating pre-trained models without fine-tuning.
  2. Supervised Fine-Tuning: Training models on RO-N3WS subsets.
  3. Natural vs. Synthetic: Comparing fine-tuning on real data vs. expressive Text-to-Speech (TTS) generated by ElevenLabs.

3. Key Contributions

1. RO-N3WS Benchmark: A high-quality, diverse Romanian dataset featuring a unique split between in-domain (news) and OOD (expressive/spontaneous) speech, addressing the lack of stylistic variation in existing corpora.
2. Comprehensive Benchmarking: A rigorous evaluation of state-of-the-art models (Whisper, Wav2Vec 2.0, and commercial APIs) under zero-shot and fine-tuned conditions, revealing specific strengths and weaknesses in domain adaptation.
3. Natural vs. Synthetic Analysis: A controlled study demonstrating that while expressive TTS can improve low-resource ASR, real human recordings provide a superior supervision signal, particularly for emotionally rich speech.
4. Open Release: The authors commit to releasing all models, scripts, and data splits to foster reproducible research in multilingual ASR.

4. Key Results

Zero-Shot Performance

• In-Domain: Commercial models (Microsoft Transcribe, Vatis) outperformed open-source models. Whisper Large (12.3% WER on ProTV) and Whisper Small + Echo (9.4% WER) were the best open-source performers.
• OOD: All models degraded significantly on OOD data, especially films (WER > 60% for some models). Whisper Large remained the most robust open-source model, but domain mismatch remained a major hurdle.

Supervised Fine-Tuning Results

• Significant Gains: Fine-tuning on RO-N3WS yielded substantial improvements. For example, Whisper Small's WER on ProTV dropped from 31.6% (zero-shot) to 4.1% after fine-tuning.
• Domain Specificity: Training on a single news source (e.g., ProTV) hurt performance on the other source (Antena 1), confirming that even within the same genre, acoustic and stylistic differences matter.
• OOD Generalization: Fine-tuning improved OOD performance, though some models showed mild overfitting to the news domain. Pre-training on the Echo dataset before fine-tuning on RO-N3WS provided consistent, albeit modest, gains in OOD robustness.

Natural vs. Synthetic Supervision

• Natural Data Superiority: Models fine-tuned exclusively on real RO-N3WS data outperformed those trained on synthetic (TTS) data across all domains.
• Synthetic Utility: Synthetic data alone improved performance over zero-shot baselines, proving its viability in low-resource scenarios.
• Hybrid Approach: A 50/50 mix of real and synthetic data narrowed the gap with natural-only models and, in some cases (e.g., film domain), surpassed natural-only performance, suggesting TTS augmentation is a viable strategy for scaling data.

5. Significance

• Advancing Low-Resource ASR: RO-N3WS provides a realistic testbed for developing robust ASR systems for Romanian, a language often underrepresented in large-scale multilingual models.
• Domain Adaptation Insights: The study highlights that "in-domain" is not monolithic; even within broadcast news, source-specific fine-tuning is necessary. It also emphasizes the importance of OOD data for evaluating true generalization.
• Synthetic Data Viability: The paper offers empirical evidence that while synthetic speech cannot fully replace human data, it is a powerful tool for augmenting low-resource datasets, provided it is high-quality and expressive.
• Community Resource: By releasing the dataset and models, the authors enable future research into dialectal diversity, noisy user-generated content, and curriculum learning strategies for Romanian and other low-resource languages.