LoASR-Bench: Evaluating Large Speech Language Models on Low-Resource Automatic Speech Recognition Across Language Families

1. Problem Statement

While Large Language Models (LLMs) have driven significant advancements in Speech Language Models (SpeechLMs), leading to strong performance in Automatic Speech Recognition (ASR) for high-resource languages, there is a critical gap in understanding their behavior in low-resource settings.

• Current Limitations: Existing benchmarks predominantly focus on high-resource languages or treat languages independently, lacking systematic analysis of typologically diverse language families (e.g., Dravidian, Uralic, Turkic) and script diversity (Latin vs. non-Latin).
• Real-World Impact: This gap hinders the deployment of SpeechLM-based ASR in real-world multilingual scenarios, as models may fail to generalize across diverse linguistic structures and writing systems, particularly when training data is scarce.
• The Need: There is an urgent need for a comprehensive benchmark to evaluate how well current SpeechLMs generalize across different language families and scripts under low-resource conditions.

2. Methodology: LoASR-Bench

The authors introduce LoASR-Bench, a new benchmark designed to systematically evaluate low-resource ASR across diverse linguistic contexts.

• Scope and Coverage:
  • Languages: 25 languages.
  • Language Families: 9 typologically diverse families (Dravidian, Romance, Uralic, Isolate, Japonic, Sino-Tibetan, Indo-Aryan, Turkic, and Koreanic).
  • Scripts: Includes both Latin and non-Latin scripts to assess cross-script generalization.
• Models Evaluated:
  • XLSR-53: A self-supervised encoder model (wav2vec 2.0 based) pre-trained on 53 languages.
  • Whisper: Whisper-Medium and Whisper-Large-v3 (encoder-decoder models trained on 680k hours of weakly supervised data).
  • Qwen Family: Multimodal SpeechLMs, specifically Qwen2-Audio-7B and the larger Qwen3-Omni-30B (covering >100 languages).
• Experimental Protocol:
  • Fine-Tuning: To ensure fair and systematic evaluation, the authors fine-tuned all models on Common Voice 16.1 data for all 25 target languages using a unified training protocol (10 epochs, batch size 8, learning rate 5e-5, mixed-precision).
  • Inference Settings: Experiments were conducted in two modes:
    1. Language-Unaware: The model receives no explicit language cue (omitting language name in prompts).
    2. Language-Aware: The model is explicitly told the target language name in the prompt.
  • Metrics: Character Error Rate (CER) for Latin scripts and Word Error Rate (WER) for non-Latin scripts.

3. Key Contributions

1. LoASR-Bench Creation: The first benchmark to systematically compare SpeechLM performance across 9 distinct language families and both Latin/non-Latin scripts, specifically targeting low-resource scenarios.
2. Systematic Evaluation: A comprehensive assessment of state-of-the-art models (XLSR-53, Whisper, Qwen), revealing significant performance variations that are often masked in single-language benchmarks.
3. Cross-Family and Cross-Script Analysis: Demonstrated that Latin-script languages consistently achieve lower error rates than non-Latin scripts, highlighting the impact of script morphology on ASR.
4. Inference Strategy Analysis: Showed that language identification (LID) is crucial; explicitly providing the language name during inference significantly improves performance, particularly for Indo-Aryan languages.
5. Unified Fine-Tuning Release: Established a reproducible set of fine-tuned checkpoints for 25 low-resource languages (including those lacking public checkpoints), with code and models to be released.

4. Key Results and Findings

• Model Performance Hierarchy:
  • Qwen Models: The Qwen family (especially Qwen3-Omni) generally outperformed Whisper and XLSR-53, particularly after fine-tuning. However, Qwen3-Omni did not universally outperform Qwen2-Audio across all families (e.g., Indo-Aryan and Turkic).
  • Whisper: Showed competitive performance but sometimes underperformed compared to Whisper-Medium in specific cases, potentially due to parameter/data mismatches.
  • Fine-Tuning Impact: Fine-tuning was critical. For example, Qwen2-Audio showed dramatic error reductions after fine-tuning on low-resource languages compared to its base model.
• Script Bias: A clear performance gap exists between Latin and non-Latin scripts. Latin-script languages (e.g., Romance family) consistently yielded lower error rates across all models, while non-Latin scripts (e.g., Dravidian, Indo-Aryan) faced higher error rates, suggesting models struggle with non-Latin morphological features.
• Model Size vs. Performance:
  • There is a negative correlation between model size and error rates (larger models perform better).
  • However, scaling from 0.04B to 30B parameters only reduced error rates from ~0.45 to ~0.32. This suggests that raw model size alone offers diminishing returns for low-resource multilingual ASR without specific fine-tuning or better data.
• Language Awareness:
  • Language-Aware Inference (providing the language name) significantly outperformed Language-Unaware inference on average.
  • Notable improvements were seen in Telugu, Finnish, and Punjabi. Conversely, some languages (e.g., Tamil) saw slight degradation, indicating that explicit cues are not a universal panacea but are generally crucial.
• Data Scarcity: Languages with extremely limited data (<1 hour) exhibited substantially higher error rates, highlighting a bottleneck for real-world deployment.

5. Significance

• Realistic Evaluation: LoASR-Bench provides a more realistic testbed for SpeechLMs by moving beyond high-resource, Latin-centric benchmarks to include typologically diverse and low-resource languages.
• Deployment Guidance: The findings indicate that current models are not yet robust enough for fully autonomous multilingual ASR in the wild. The reliance on explicit language cues suggests that future systems must integrate Language Identification (LID) directly into the ASR pipeline.
• Resource Allocation: The study highlights that simply increasing model size is insufficient; target-language fine-tuning and addressing data scarcity (especially for non-Latin scripts) are more critical for improving low-resource ASR.
• Community Resource: By releasing fine-tuned checkpoints for 25 languages, the authors provide a unified baseline for future research, enabling reproducible comparisons across the low-resource ASR landscape.