Technical Summary: Linguistically Informed Evaluation of Multilingual ASR for African Languages

Problem Statement

Standard evaluation metrics for Automatic Speech Recognition (ASR), particularly Word Error Rate (WER), fail to accurately characterize model performance for African languages. WER treats all errors as equivalent lexical failures, collapsing distinct phonological, tonal, and segmental errors into a single score. This "all-or-nothing" approach obscures the model's actual learning progress, especially in low-resource and tonal languages where a single tone or segment error can render a word incorrect despite the model correctly identifying the majority of phonetic features. Consequently, WER often yields near-total error rates (e.g., >99%) for endangered languages, masking the fact that models may be successfully learning acoustic-to-phonetic mappings.

Methodology

The authors evaluate multilingual ASR models on two typologically related African languages: Yoruba (a low-resource language present in pretraining data) and Uneme (an endangered Edoid language absent from pretraining data).

Data

• Yoruba: Sourced from the FLORES dataset (FLEURS corpus), comprising over 10 hours of audio-transcript pairs.
• Uneme: A newly created dataset from 2025 fieldwork in Edo State, Nigeria. It includes approximately 8 hours of recordings across narratives, interviews, and elicitation. Crucially, the authors created two distinct test sets: Test-natural (conversational speech) and Test-careful (slow, deliberate, word-by-word reading) to investigate domain shift.
• Baseline: English data from FLORES was used for high-resource comparison.

Models and Architecture

The study employs three speech encoders:

1. mHuBERT-25-Hz: Based on HuBERT-base-95M with additional English data.
2. mHuBERT-147: A multilingual model trained on 147 languages.
3. XLS-R: A wav2vec 2.0 variant trained on over 100 languages.

Two decoder architectures were tested:

• Linear Decoder: A single fully connected layer to test encoder capability in isolation.
• Transformer Decoder: A two-block Transformer model with Rotary Embeddings, SWiGLU activation, and RMSNorm.

Evaluation Metrics

Beyond standard WER and Character Error Rate (CER), the authors introduce a linguistically informed framework:

• Grapheme-to-Phoneme (G2P) Conversion: Orthography is mapped to International Phonetic Alphabet (IPA) segments.
• Phonological Feature Vectors: Segments are mapped to 24-dimensional sparse vectors (values: -1, 0, +1) representing articulatory features (e.g., voicing, place, manner).
• Feature Error Rate (FER): Calculates error based on the mismatch of active phonological features using a Needleman-Wunsch alignment with a modified cost function that ignores non-applicable features (e.g., tone for English).
• Tone Error Rate (TER): A specific extension of FER that isolates suprasegmental tone features (High, Low, Mid, Downstep) anchored to vowels and syllabic nasals.

Key Results

1. Discrepancy Between WER and Linguistic Metrics

The study demonstrates that WER systematically mischaracterizes performance.

• Yoruba: The best model achieved a WER of 0.788, but a FER of only 0.151. This indicates that while lexical accuracy is low, the model correctly predicts roughly 85% of the underlying phonological features.
• Uneme: For the unseen language, the best model achieved a near-total WER of 0.997. However, the FER was 0.267, meaning nearly 74% of phonological features were correctly predicted. The TER was 0.402, highlighting that tone errors are the primary driver of lexical failure.

2. Segmental vs. Suprasegmental Performance

Models consistently perform better on segmental features (consonants and vowels) than on suprasegmental features (tones).

• Tone Challenges: In both Yoruba and Uneme, Tone Error Rates (TER) were significantly higher than overall FER.
  • Yoruba: Mid tones were the most challenging (FER ≈ 0.34).
  • Uneme: Downstep (unmarked tone) was the most challenging category, often tied with or exceeding High tone errors.
• Vowel Quality: Back vowels and diphthongs showed higher error rates in English and Uneme, suggesting difficulty in modeling spectral dynamics and vowel backness contrasts.

3. The "Careful Speech" Paradox

Contrary to intuition, models performed significantly worse on the Test-careful dataset compared to Test-natural.

• Uneme Results: The best model's FER degraded from 0.267 (natural) to 0.515 (careful).
• Attribution: The authors attribute this to domain shift. The models were fine-tuned on natural, conversational data. The "careful" speech, characterized by exaggerated pauses and altered prosody, represented an out-of-distribution shift that the models failed to generalize to.

4. Decoder Architecture Impact

Transformer decoders significantly outperformed Linear decoders across all languages. Linear decoders often collapsed into degenerate outputs (WER ≈ 1.0), confirming that simple linear projections are insufficient for mapping acoustic embeddings to discrete tokens, particularly for tonal languages requiring complex feature integration.

Significance and Claims

The paper claims that Feature Error Rate (FER) and Tone Error Rate (TER) provide a more nuanced and accurate assessment of ASR progress for African languages than WER or CER.

• Revealing Partial Success: FER reveals that models are learning phonological structures even when lexical accuracy is near zero, a critical insight for endangered language documentation where WER is often 100%.
• Diagnostic Utility: By isolating errors to specific features (e.g., tone vs. segment), FER guides data collection and model improvement efforts, identifying that tone and specific vowel qualities are the primary bottlenecks.
• Methodological Contribution: The work provides the first ASR baseline for Uneme and demonstrates that active collaboration between field linguists and NLP practitioners is necessary to create evaluation frameworks that respect the linguistic typology of African languages.

The authors conclude that while phonological reconstruction is feasible for endangered languages, bridging the gap to lexical correctness requires addressing suprasegmental challenges (specifically tone) and managing domain shifts in speech styles.