Exploiting Domain-Specific Parallel Data on Multilingual Language Models for Low-resource Language Translation

1. Problem Statement

Neural Machine Translation (NMT) systems based on Multilingual Sequence-to-Sequence Language Models (msLMs), such as mBART, often fail to deliver expected performance for Low-Resource Languages (LRLs) when the available parallel data is scarce or when the target domain is underrepresented in the pre-training corpus.

The core challenge addressed is how to effectively utilize auxiliary domain parallel data (data from different domains than the target) to build domain-specific NMT systems for LRLs. Specifically, the authors investigate two primary scenarios:

1. In-domain: The target domain has some parallel data, but it is limited.
2. Out-domain: The target domain has no parallel data for training (zero-shot scenario), requiring reliance entirely on auxiliary data.

The paper seeks to determine the optimal strategy for integrating this auxiliary data: should it be used for continuous pre-training (bitext denoising) or fine-tuning (single-stage or multi-stage transfer learning)? Furthermore, it investigates how domain divergence (the semantic and lexical distance between source and target domains) impacts these strategies.

2. Methodology

2.1 Experimental Setup

• Models: The primary baseline is mBART (multilingual BART). An ablation study was conducted using NLLB-200-distilled-600M.
• Languages: Five LRLs were selected: Sinhala (si), Tamil (ta), Kannada (ka), Gujarati (gu), and Hindi (hi), paired with English. Kannada was treated as a special case as it is not natively supported in mBART, requiring related-language fine-tuning (using Telugu).
• Datasets: Parallel corpora from four distinct domains were used:
  • CC (CommonCrawl): Open-domain, noisy, large volume.
  • Bible: Religious domain, clean, available for all languages.
  • PMI/Gvt: News and Government/Administrative domains.
  • FLORES: Used exclusively for out-domain testing.
• Metrics: SentencePiece BLEU (spBLEU) and Jenson-Shannon Divergence (JSD) to quantify domain divergence.

2.2 Techniques Evaluated

The authors systematically compared four main strategies for utilizing auxiliary data:

1. Continuous Pre-training: Further pre-training the msLM using bitext denoising objectives (with or without monolingual denoising) before fine-tuning.
2. Fine-Tuning Strategies:
   • Vanilla FT (Single-domain): Fine-tuning on a single domain (baseline).
   • Multi-domain FT: Fine-tuning once on a mixture of data from multiple domains.
   • Single-domain ITTL (Intermediate Task Transfer Learning): Two-stage fine-tuning. Stage 1: Fine-tune on one auxiliary domain. Stage 2: Fine-tune on the target domain.
   • Multi-domain ITTL: Two-stage fine-tuning. Stage 1: Fine-tune on a mixture of multiple auxiliary domains. Stage 2: Fine-tune on the target domain.

2.3 Experimental Variables

• Data Sizes: Varied from 1k to 25k sentences (and up to 100k for CC) for both intermediate and final stages.
• Domain Combinations: Tested various pairings of CC, Bible, and PMI/Gvt to measure the impact of domain divergence.
• Ablation Studies: Included up-sampling smaller datasets to match larger ones and testing with different msLMs (NLLB).

3. Key Contributions & Findings

3.1 Continuous Pre-training is Ineffective for Small Data

The study found that continuous pre-training with bitext denoising yields no significant gains when the auxiliary parallel data size is small (≤25k sentences). In some cases (e.g., for Kannada), it even degraded performance compared to standard fine-tuning. The authors conclude that pre-training is not a viable strategy for LRLs unless massive amounts of monolingual data are also available, which was outside the scope of this study.

3.2 Impact of Data Size on Fine-Tuning Strategies

The optimal strategy is heavily dependent on the size of the target domain data:

• Small Data (1k sentences):
  • In-domain: Multi-domain Fine-Tuning (Multi-FT) is the most efficient and effective option, outperforming complex ITTL setups with minimal computational cost.
  • Out-domain: Multi-FT and Multi-domain ITTL both outperform Vanilla FT, but Multi-FT is more stable.
• Medium Data (Intermediate stage 25k, Final stage 1k):
  • In-domain: Multi-domain ITTL becomes the best-performing technique, confirming previous observations that a large intermediate task helps when the final task is small.
• Large Data (Target domain ≥25k sentences):
  • In-domain: When the target domain has sufficient data (approx. 25k), Vanilla FT (fine-tuning only on the target domain) is usually sufficient. Adding auxiliary data via ITTL or Multi-domain mixing provides negligible or no gains.
  • Out-domain: Auxiliary strategies still help, but the gains diminish as target data availability increases.

3.3 The Critical Role of Domain Divergence

• Divergence Matters: The performance of ITTL and Multi-domain strategies is highly sensitive to the JSD (divergence) between the auxiliary and target domains.
• Selection Strategy: For out-domain testing, the best results are achieved when the final stage of fine-tuning uses data from a domain closest to the target test set (e.g., if testing on FLORES, using PMI/Gvt for the final stage is better than using Bible, as FLORES is closer to news/gov data).
• Mixing Limits: Blindly mixing data from more than two domains (e.g., CC + Bible + PMI) often leads to performance drops due to conflicting domain distributions. The paper advises against mixing more than two domains unless their divergence is carefully managed.

3.4 Stability and Efficiency

• Multi-domain FT vs. Multi-domain ITTL: While Multi-domain ITTL can achieve slightly higher peaks in specific "large-small" data scenarios, Multi-domain FT is generally more robust. It exhibits lower variance in performance across different domain combinations and is computationally more efficient (single-stage vs. two-stage).
• Recommendation: For out-domain scenarios, Multi-domain FT is preferred because it is less susceptible to domain divergence than Multi-domain ITTL.

4. Significance and Recommendations

This paper provides a crucial "rule of thumb" for researchers and practitioners working with LRLs who lack extensive computational resources to run exhaustive experiments.

Key Recommendations:

1. Avoid Pre-training: Do not use bitext denoising pre-training if you only have small parallel datasets (<25k).
2. In-domain Strategy:
   • If target data is <10k: Use Multi-domain FT.
   • If target data is 1k but you have a large auxiliary set (25k+): Use Multi-domain ITTL.
   • If target data is ≥25k: Stick to Vanilla FT; auxiliary data adds little value.
3. Out-domain Strategy: Calculate domain divergence (JSD) first. Select auxiliary data that is closest to the target domain. Use Multi-domain FT for a better trade-off between performance and stability.
4. Data Mixing: Do not blindly combine data from three or more domains; the noise introduced by high divergence often outweighs the benefits of increased data volume.

Conclusion

The study demonstrates that while msLMs offer a strong foundation for LRL translation, the method of adapting them to specific domains is non-trivial. The "best" method is not universal; it is a function of data volume and domain similarity. By prioritizing Multi-domain Fine-Tuning for small datasets and Vanilla Fine-Tuning for larger target datasets, researchers can achieve optimal results without the computational overhead of complex pre-training or multi-stage transfer learning pipelines.