Tone-Conditioned Curriculum Learning for Low-Resource Bantu Speech Recognition

Technical Summary: Tone-Conditioned Curriculum Learning for Low-Resource Bantu Speech Recognition

Problem Statement
Southern Bantu languages (e.g., isiZulu, isiXhosa, Sesotho, Setswana, Tshivenda, Xitsonga) are spoken by over 80 million people but remain underrepresented in Automatic Speech Recognition (ASR). Current foundation models, such as Whisper and MMS, produce zero-shot Word Error Rates (WER) exceeding 100% on these languages, rendering them impractical for education and public services. A primary challenge is that these languages rely heavily on tone and prosody to encode grammatical relations and lexical meaning, yet standard orthographies omit tone. Unlike Mandarin, where tone is often syllable-local, Southern Bantu systems exhibit high tone spreading and phrasal contours. Standard curriculum learning approaches, which typically order data by recognition error (WER) or duration, fail to account for these specific phonological and morphotonal complexities.

Methodology
The authors propose a Tone-Conditioned Curriculum Framework designed to address data scarcity and tonal complexity through three integrated phases:

1. Hybrid Difficulty Scoring:
   Instead of relying solely on recognition error, the authors define a hybrid difficulty score $s(u)$ for each utterance $u$:
   $$s(u) = \alpha \cdot \text{WER}_{\text{norm}}(u) + \beta \cdot \text{Tonal}_{\text{norm}}(u)$$
   With weights $\alpha=0.7$ and $\beta=0.3$, this metric balances empirical model performance with linguistic complexity. The tonal component aggregates morphotonal features (transition density, unique pattern variety) extracted from F0 contours without requiring manual phonetic transcription. Utterances are mapped to difficulty quintiles to drive staged training.

2. Tone-Conditioned Gated Adapters:
   To inject tonal context into the model, the authors introduce lightweight, gated adapters (adding ~2.1M parameters, or 0.3% of the base model). A 2-layer MLP maps a 5-dimensional tonal vector (derived from F0 statistics like transition rate and range) to gate values. These gates modulate the encoder representations, allowing the model to dynamically increase adapter contribution for utterances with complex tonal profiles while retaining pretrained representations for simpler ones.

3. Staged Curriculum Training:
   Training follows a three-stage schedule inspired by CL-DM to prevent overfitting on noisy, difficult samples early in training:

   • Stage 1: Trains on the easiest 40% of samples.
   • Stage 2: Expands to the easiest 80%.
   • Stage 3: Utilizes the full training set.
     This progressive merging aims to preserve performance on stable patterns while adapting to harder tonal structures.

Key Contributions

1. Hybrid Scoring Function: A novel difficulty metric combining WER with morphotonal features specifically tailored for Bantu languages.
2. Gated Tone Adapters: A parameter-efficient mechanism (2.1M parameters) that conditions encoder representations on per-utterance tonal statistics.
3. Comprehensive Evaluation: An analysis of three foundation architectures (Whisper, W2V-BERT, MMS) across two datasets (Swivuriso and NCHLT), revealing distinct architecture-language preferences.
4. Generalization Analysis: Evidence that model transferability varies significantly by language family, with W2V-BERT showing superior transfer for Nguni languages compared to Whisper.

Results
The study evaluated 12 configurations across the three model families on six Southern Bantu languages:

• Baseline Performance: Zero-shot baselines (Whisper, MMS) failed with WERs >100%. Fine-tuning reduced errors substantially, but no single model outperformed others across all languages.
• Architecture Divergence:
  • Nguni Languages (isiZulu, isiXhosa): W2V-BERT significantly outperformed Whisper (e.g., 24.79% vs. 28.12% WER on Swivuriso for isiZulu).
  • Sotho-Tswana Languages (Sesotho, Setswana): Whisper outperformed W2V-BERT (e.g., Setswana reached 18.60% WER with Whisper Tone+Curr.).
• Impact of Tone Conditioning:
  • W2V-BERT: Tone conditioning provided a significant relative gain (7.2% reduction in average WER from 25.65% to 23.81%), particularly benefiting CTC decoding.
  • Whisper & MMS: Tone conditioning offered minimal or mixed benefits. For MMS, curriculum ordering based on WER alone was more effective than the hybrid approach.
• Transfer Robustness: Performance varied when transferring from the community-recorded Swivuriso dataset to the studio-recorded NCHLT dataset. For instance, Tshivenda showed a large performance gap (22 WER points) between datasets, whereas Xitsonga demonstrated stronger stability with W2V-BERT tone conditioning.

Significance and Claims
The paper claims that no single model architecture is universally optimal for all six Southern Bantu languages. The suitability of an architecture is contingent on the language family (Nguni vs. Sotho-Tswana). Consequently, the authors argue that reliable deployment requires:

1. Language-specific model selection rather than a one-size-fits-all approach.
2. Validation across corpora to account for distribution shifts between community and studio recordings.
3. Integration of tonal structure into adaptation strategies, demonstrating that parameter-efficient adaptation can leverage linguistic structure without manual phonological annotation.

The work highlights that while curriculum learning and tone conditioning improve robustness, their effectiveness is highly dependent on the interaction between the specific architecture and the linguistic properties of the target language.