TG-ASR: Translation-Guided Learning with Parallel Gated Cross Attention for Low-Resource Automatic Speech Recognition

1. Problem Statement

The paper addresses the critical challenge of Low-Resource Automatic Speech Recognition (ASR), specifically focusing on Taiwanese Hokkien.

• The Data Scarcity Issue: While vast amounts of spoken content exist in Taiwanese Hokkien (e.g., TV dramas, online videos), high-quality transcriptions are scarce. Most available subtitles are in Mandarin, creating a misalignment between the spoken language (Taiwanese Hokkien) and the available text supervision (Mandarin).
• Limitations of Existing Methods: Traditional approaches like data augmentation, multilingual joint training, or transfer learning from high-resource languages often fail when the target language is typologically distinct and data-sparse. They struggle with unstable convergence and reduced accuracy when the linguistic gap is large.
• The Goal: To develop an ASR system that can generate accurate Taiwanese Hokkien transcriptions by leveraging the abundant, aligned Mandarin subtitles found in media, effectively using translation as a supervisory signal.

2. Methodology: TG-ASR Framework

The authors propose TG-ASR (Translation-Guided ASR), a framework designed to integrate multilingual translation embeddings into the ASR decoding process. The system is built upon the Whisper model and operates in a two-stage training process.

A. Core Architecture

1. Backbone: The system uses Whisper (Small) as the base encoder-decoder model.
2. Two-Stage Training:
   • Stage 1: Fine-tune the entire Whisper model on the target language (Taiwanese Hokkien) to adapt acoustic representations.
   • Stage 2: Freeze the Whisper encoder and the initial decoder parameters. Introduce and train the new Parallel Gated Cross-Attention (PGCA) layers.
3. Multilingual Embedding Extraction:
   • Auxiliary language transcriptions (e.g., Mandarin, English, Spanish) are generated using SeamlessM4T (a multilingual translation model).
   • These translated texts are encoded using a pre-trained Multilingual BERT (mBERT) to produce contextualized translation embeddings ($E_l$).
   • The mBERT model remains frozen during training.

B. The Parallel Gated Cross-Attention (PGCA) Mechanism

This is the novel contribution of the paper. Instead of a single cross-attention module, PGCA introduces multiple parallel cross-attention branches, one for each auxiliary language.

• Mechanism: For each decoder input token, the model computes cross-attention between the decoder state and the embeddings of $L$ different auxiliary languages.
• Gating: A learnable tanh gating mechanism ($\alpha_{attn}$) dynamically regulates the contribution of each auxiliary language.
  • Formula: $Y' = Y + \sum_{l=1}^{L} \tanh(\alpha_{attn}^{(l)}) \times \text{attn}(Y, E_l, E_l)$
  • This allows the model to adaptively promote helpful languages (high semantic alignment) and suppress noisy or less relevant ones.
• Integration: The weighted embeddings are aggregated and added to the decoder input before passing through a Feedforward Neural Network (FNN).

3. Key Contributions

1. TG-ASR Framework: A novel architecture that utilizes translation-guided learning via the PGCA mechanism to integrate multilingual semantic cues without disrupting the core decoding process.
2. YT-THDC Corpus: The release of a new dataset, YT-THDC (YouTube-sourced Taiwanese Hokkien Drama Corpus).
   • Size: ~30 hours of speech.
   • Content: 55,843 utterances from Taiwanese Hokkien dramas.
   • Alignment: Meticulously aligned with existing Mandarin subtitles and manually verified Taiwanese Hokkien transcriptions.
3. Comprehensive Analysis: A deep investigation into which auxiliary languages are most effective, how the gating mechanism behaves, and the impact of translation quality.

4. Experimental Results

Experiments were conducted on the YT-THDC corpus using Character Error Rate (CER) as the primary metric.

• Performance Gains:

  • The baseline model (trained only on Taiwanese Hokkien) achieved a CER of 13.40%.
  • Using Mandarin (Ground Truth) as the sole auxiliary language reduced CER to 11.87% (11.42% relative reduction).
  • The optimal configuration using Mandarin + Spanish (via PGCA) achieved the lowest CER of 11.42%, representing a 14.77% relative reduction compared to the baseline.
  • Notably, the multilingual combination outperformed the single best language (Mandarin), suggesting complementary benefits.

• Ablation Studies:

  • Gating: Removing the tanh gating mechanism caused a slight performance drop, confirming its role in regulating information flow.
  • Parallel vs. Sequential: Parallel attention significantly outperformed sequential attention, proving that concurrent access to all languages is superior.
  • Fusion Strategy: Simple addition or concatenation of embeddings resulted in massive performance degradation (CER > 24%), highlighting the necessity of the gated cross-attention mechanism.

• Language Analysis:

  • Proximity vs. Performance: While languages with higher semantic proximity to Taiwanese Hokkien (measured via mBERT [CLS] similarity) generally performed better, the relationship was not strictly monotonic (e.g., Hindi had high proximity but poor performance).
  • Gating Behavior: The model learned to assign positive weights to high-performing languages (Mandarin, Spanish) and negative weights to less effective ones (English, Hindi), actively filtering out noise.
  • Translation Source: Translations generated by SeamlessM4T yielded better results than those from NLLB, indicating that translation quality directly impacts ASR guidance.

5. Significance and Conclusion

• Practical Application: TG-ASR demonstrates a viable path to preserving and digitizing low-resource languages by leveraging existing media with mismatched subtitles (e.g., using Mandarin subtitles to learn Taiwanese Hokkien speech).
• Robustness: The PGCA mechanism proves robust against translation noise and varying language proximities by adaptively weighting inputs.
• Scalability: The framework shows that adding multiple auxiliary languages can improve performance, provided the gating mechanism can manage the interference.
• Future Work: The authors aim to remove the reliance on auxiliary text during inference (via knowledge distillation) and expand the approach to other underrepresented languages.

In summary, this paper establishes that translation-guided learning, when coupled with a gated cross-attention mechanism, is a highly effective strategy for overcoming data scarcity in low-resource ASR, achieving state-of-the-art results on Taiwanese Hokkien without requiring massive amounts of native transcribed data.