Technical Summary: KIT's Low-resource Speech Translation Systems for IWSLT2025

Problem Statement

The paper addresses the challenges of building Speech Translation (ST) systems for low-resource languages, specifically focusing on three language pairs: Bemba, North Levantine Arabic, and Tunisian Arabic into English. A primary obstacle in this domain is the scarcity of end-to-end (E2E) parallel ST training data. While Automatic Speech Recognition (ASR) and Machine Translation (MT) resources are more accessible, the lack of direct ST data hinders the training of robust E2E models. Additionally, the paper notes that low-resource ST systems often suffer from imbalanced parameter usage, necessitating effective regularization strategies to improve generalization without relying on extra external resources.

Methodology

The authors propose a comprehensive framework combining synthetic data augmentation, model regularization, and system combination strategies, built upon multilingual pre-trained models.

1. Synthetic Data Augmentation

To mitigate data scarcity, the authors explore two distinct augmentation modalities:

• MT-augmented ST: Translations are generated from existing ASR datasets using trained MT models to create synthetic ST pairs. This approach was particularly applied to North Levantine, which lacks parallel ST training data.
• TTS-augmented ST: Text-to-Speech (TTS) models (specifically E2TTS and VITS) are trained on ASR data to synthesize speech from MT datasets. This generates synthetic speech-text pairs to augment ST training data, primarily tested on Bemba.

2. Model Regularization (Intra-Distillation)

The study extends the application of Intra-Distillation (ID), a technique previously successful in low-resource MT, to ASR, MT, and ST tasks. Unlike previous works that combine task objectives and ID loss during a single fine-tuning stage, the authors employ a two-stage approach:

1. Vanilla Fine-tuning: Adapting the pre-trained model to the downstream task.
2. ID Fine-tuning: Regularizing the adapted model using its own intermediate predictions.
   This strategy aims to retain task-specific adaptation benefits while unlocking robustness through regularization.

3. System Combination

The authors combine cascaded (ASR + MT) and end-to-end ST systems using Minimum Bayes Risk (MBR) decoding. By treating hypotheses from both system types as pseudo-references and using BLEU as the utility function, they aim to leverage the strengths of both architectures.

4. Pre-trained Models and Resources

The experiments utilize several state-of-the-art pre-trained models:

• SeamlessM4T: A highly multilingual, multimodal model (v2 large).
• NLLB: A multilingual MT model (1.3B parameters).
• MMS and XEUS: Multilingual ASR models.
  The authors leverage both development set data and extensive external resources (including LDC datasets, Tatoeba, UFAL, and internally developed conversational data) under the unconstrained track conditions.

Key Results

The experimental results across the three language pairs yield several significant findings:

• Synthetic Data Efficacy:
  • For North Levantine, a system trained solely on synthetic data (generated via MT-augmentation) slightly surpassed the cascaded system trained on real data, achieving a ~1 BLEU point improvement.
  • For Bemba, TTS-augmented data improved ST performance by up to 1 BLEU point. However, for North Levantine, relying exclusively on TTS-generated data resulted in lower performance compared to the cascaded system, attributed to the under-developed state of the TTS model for that specific dialect.
• Model Regularization: Intra-distillation (ID) consistently improved performance across ASR, MT, and ST tasks for most models, yielding approximately a 1-point reduction in Word Error Rate (WER) for ASR and a 1 BLEU point gain for MT and ST. An exception was noted where ID negatively impacted ASR performance for Arabic dialects.
• System Combination: Applying MBR decoding to individual systems yielded minimal to no improvement. However, combining the cascaded and E2E systems via MBR decoding consistently resulted in an improvement of approximately 1.5 BLEU points.
• Language-Specific Variance: The effectiveness of pre-trained models and training strategies varied significantly across languages. For instance, NLLB outperformed SeamlessM4T on Bemba, while the reverse was not necessarily true for Arabic dialects without specific fine-tuning strategies.

Significance and Claims

The paper claims that its primary significance lies in demonstrating that synthetic data augmentation and model regularization are effective strategies for enhancing low-resource speech translation without requiring additional real-world resources.

Specifically, the authors conclude:

1. Synthetic data is promising for improving model performance, provided the generated data is of reasonable quality.
2. Model regularization (specifically Intra-Distillation) is a generalizable approach that enhances performance across different tasks (ASR, MT, ST) and pre-trained models.
3. Language-specific approaches are critical. The divergent findings across Bemba and the Arabic dialects highlight that a "one-size-fits-all" strategy is insufficient; the choice of pre-trained models and training strategies must be tailored to the specific language and corpus characteristics.

The work reinforces the practical viability of leveraging separate ASR and MT components to build cascaded systems and the potential of synthetic data to bridge gaps where parallel ST data is non-existent.