Task Arithmetic with Support Languages for Low-Resource ASR

1. Problem Statement

The paper addresses the challenge of Automatic Speech Recognition (ASR) for low-resource languages, specifically within the context of the 2025 Mozilla Common Voice Spontaneous Speech ASR shared task.

• Data Scarcity: Many low-resource languages lack sufficient labeled training data, making standard supervised training ineffective.
• Domain Mismatch: The task focuses on spontaneous speech, whereas many existing benchmarks and pre-trained models are optimized for scripted speech.
• Model Limitations: While foundation models like OpenAI's Whisper are multilingual, their performance varies significantly across languages due to training data dominance by high-resource languages. Furthermore, large models often struggle to adapt effectively with very limited data compared to smaller, more targeted models.
• Goal: To improve ASR performance for 26 low-resource languages (21 with training data, 5 unseen) by leveraging data from genetically related "support languages" without requiring massive computational resources for full retraining.

2. Methodology

The authors propose a framework combining Task Arithmetic with Low-Rank Adaptation (LoRA) to merge models trained on different languages.

A. Model Architecture and Training

• Base Models: The study utilizes two variants of the Whisper model:
  • Whisper-tiny: Fully fine-tuned for all target languages.
  • Whisper-large-v3: Fine-tuned using LoRA adapters (to save memory and parameters). This model was quantized to 4-bits due to resource constraints.
• Task Vectors (TVs): Following Ilharco et al. (2023), a "task vector" is defined as the difference between the weights of a fine-tuned model and the pre-trained base model ($\tau = \theta_{fine-tuned} - \theta_{base}$).
• Support Language Strategy:
  • For each target language, the authors identify genetically related "support languages" (e.g., using 13 Quechuan languages to support Puno Quechua).
  • Separate models (or LoRA adapters) are trained on these support languages using Common Voice (CV) data.
  • Proxy Language Initialization: To maximize token overlap, the language token during training is fixed to a "proxy language" (a pre-training language sharing the same script as the target) rather than the target language itself.

B. Task Arithmetic Integration

The core innovation is the linear combination of task vectors to create a final model:
$$\theta_{final} = \theta_{target} + \lambda \tau_{support}$$

• $\theta_{target}$: Weights of the model fine-tuned on the target language.
• $\tau_{support}$: The task vector derived from the support language(s).
• $\lambda$: A scaling hyperparameter optimized via Bayesian optimization on the target language's validation set (range $[0, 1]$).
• For LoRA: The process involves scaling the weights of the support adapter and adding them to the target adapter's weights before applying them to the base model.

C. Data Processing

• Data Sources: Scripted speech from Common Voice for support languages and unseen targets; spontaneous speech for the 21 trained targets.
• Quality Filtering: Samples with zero-length audio, empty transcripts, or flagged as problematic were removed. Audio >30s was truncated.
• Upsampling: To prioritize high-quality transcriptions, samples were upsampled based on "votes" (upvotes minus downvotes) in the Common Voice metadata.

3. Key Contributions

1. Extension of Task Arithmetic to ASR: The paper expands the application of task arithmetic from NLP to ASR, specifically demonstrating its utility in combining models across different languages rather than just different tasks.
2. Support Language Integration: It introduces a systematic method to identify and integrate genetically related support languages to boost low-resource performance, moving beyond the limited scope of previous work (which was restricted to 4 languages present in Whisper's training data).
3. LoRA-based Task Arithmetic: The authors successfully demonstrate that task arithmetic can be performed via linear combinations of LoRA layers, not just full parameter task vectors.
4. Empirical Analysis of Model Size: The study provides evidence that for low-resource settings with limited data, smaller models (Whisper-tiny) often outperform larger models (Whisper-large-v3) when combined with task arithmetic, challenging the assumption that larger models are always superior in data-scarce regimes.

4. Results

The system was evaluated across 23 low-resource target languages using Word Error Rate (WER).

• Whisper-tiny Performance:
  • Incorporating support language task vectors improved all applicable target models.
  • Average WER Improvement: -3.51%.
  • Maximum Improvement: Up to -9.93% for specific languages.
  • The approach acted as a safety net; even in worst-case scenarios, performance did not degrade below the baseline of using only the target language model.
• Whisper-large-v3 Performance:
  • Generally performed worse than Whisper-tiny and the task baseline for most languages, likely due to suboptimal hyperparameters and limited convergence of LoRA adapters.
  • However, it still showed average WER improvements of -5.43% when support languages were added.
  • Adyghe Case Study: The method improved the Adyghe model by -49.72%, highlighting the potential for massive gains when the baseline is weak.
• Correlation Analysis:
  • For Whisper-tiny, a moderate positive correlation (Pearson $r=0.52$) was found between token overlap similarity (cosine similarity of token counts) and WER improvement. This suggests that more linguistically similar support languages yield better results.
  • For Whisper-large-v3, no significant correlation was found between similarity and improvement, likely due to incomplete adapter training. Instead, improvement correlated strongly with the baseline WER (larger gains where the baseline was worse).

5. Significance and Limitations

Significance:

• Efficiency: The approach offers a computationally efficient way to leverage existing data from related languages without retraining massive foundation models from scratch.
• Indigenous Language Support: The method is particularly valuable for indigenous and low-resource languages where data is scarce, potentially leading to more accessible technology for these communities.
• Model Selection: It challenges the "bigger is better" paradigm, suggesting that targeted fine-tuning of smaller models with task arithmetic is a superior strategy for low-resource ASR.

Limitations:

• Hyperparameter Search: Due to compute constraints, the search for optimal training hyperparameters and the scaling factor $\lambda$ was not exhaustive.
• Base Model Choice: The authors note that the Massively Multilingual Speech (MMS) model might be a better base model than Whisper for low-resource tasks, as the task baseline using MMS outperformed their systems.
• Context Window: The 30-second input limit of Whisper negatively impacted performance on longer spontaneous speech recordings.
• Support Language Selection: The current selection of support languages relied on genetic relatedness and script availability; future work could utilize phonological or lexical overlap metrics for better selection.

In conclusion, the paper demonstrates that Task Arithmetic is a robust technique for enhancing low-resource ASR, particularly when combining smaller, efficiently fine-tuned models with data from genetically related languages.