Technical Summary: Modular Monolingual Adaptation using Pretrained Language Models

Problem Statement

The adaptation of Pretrained Multilingual Language Models (PMLMs) to low-resource languages typically involves full model finetuning on the target language. However, this approach faces significant challenges in low-resource settings:

1. Overfitting: Training the entire parameter space, particularly the large embedding matrices (which constitute ~25% of parameters), on limited data often leads to overfitting.
2. Curse of Multilinguality: In fixed-capacity multilingual models, high-resource languages dominate the parameter space, potentially limiting performance for underrepresented languages.
3. Inefficiency: Full finetuning is computationally expensive and requires substantial VRAM and training time.
4. Vocabulary Mismatch: Reusing a multilingual tokenizer for an unseen language can result in high token fertility (splitting words into many subwords), increasing computational costs and reducing efficiency.

While techniques like Language Adaptive Finetuning (LAFT) and vocabulary extension exist, the authors hypothesize that full model tuning is often unnecessary and that a more modular approach could yield better results with fewer resources.

Methodology

The authors propose a Modular Monolingual Adaptation framework that decouples the tokenizer and embeddings from the transformer layers. The core components are:

1. Custom Tokenization: A new tokenizer is trained specifically on the target language's corpus using the WordPiece algorithm, maintaining a vocabulary size of 30k. This replaces the original multilingual tokenizer.
2. Embedding Replacement and Freezing:
   • The input and output embeddings of the PMLM are replaced with a new matrix corresponding to the custom vocabulary.
   • Freezing: These new embedding layers are frozen during training.
   • Initialization Strategies: The authors experiment with three methods to initialize these new embeddings:
     • Model-based: Re-tokenizing the original vocabulary and averaging the weights of the original tokens that compose the new tokens.
     • FastText: Training static embeddings on the target corpus using FastText.
     • Random: Random initialization.
3. Selective Parameter Training: Only the non-embedding parameters (the Transformer layers and the LM Head) are updated using the standard Masked Language Modeling (MLM) objective.
4. Experimental Setup:
   • Languages: Scottish Gaelic (gd), Irish (ga), and Quechua (qu). Quechua is noted as a very low-resource language with only ~8.5k training instances.
   • Models: BERT, Multilingual BERT (mBERT), and Multilingual ModernBERT (mmBERT).
   • Tasks: Mask-filling (MLM), Named Entity Recognition (NER), and Part-of-Speech (POS) tagging.
   • Baselines: Compared against full finetuning, embedding-only training, and Low-Rank Adaptation (LoRA).

Key Contributions

• Modular Framework: A novel adaptation strategy that replaces the tokenizer, freezes embeddings, and tunes only the remaining parameters.
• Comprehensive Analysis: A systematic evaluation of the impact of tokenizers, embedding initialization strategies, and model architectures (BERT vs. mBERT vs. mmBERT) on monolingual adaptation.
• Efficiency Gains: Demonstration that this approach significantly reduces the number of trainable parameters (e.g., ~50% reduction for mBERT) and training time while maintaining or improving performance.

Results

The experiments yielded the following findings:

1. Non-Embedding Training vs. Full Tuning:

   • On the MLM (mask-filling) task, the proposed non-embedding training strategy consistently outperformed full finetuning across all languages and models. The authors attribute this to the regularizing effect of freezing embeddings, which prevents the model from overfitting to the small training set.
   • On downstream tasks (NER and POS), non-embedding training performed comparably to full finetuning and significantly better than embedding-only training. This suggests that while embeddings are important, the Transformer layers are sufficient to learn task-specific contextual representations when the embedding space is fixed.

2. Impact of Tokenizers:

   • Using a custom tokenizer consistently outperformed reusing the multilingual tokenizer. This improvement is attributed to reduced token fertility and better sub-word segmentation for the target language.
   • The custom vocabulary also reduced the total model size and training memory requirements.

3. Embedding Initialization:

   • In the non-embedding training setting, the choice of embedding initialization (Model-based, FastText, or Random) had a minor impact on performance. Randomly initialized embeddings performed nearly as well as pretrained ones, suggesting the Transformer layers can effectively realign to the new lexical space.
   • Conversely, under full finetuning, default embeddings sometimes hindered performance, likely because the model had to "unlearn" irrelevant representations from the pretraining data.

4. Model Architecture:

   • Multilingual models (mBERT, mmBERT) generally outperformed monolingual BERT and randomly initialized transformers, demonstrating effective cross-lingual knowledge transfer even when the target language was minimally represented in pretraining data.
   • mmBERT showed superior MLM performance due to its larger capacity but occasionally underperformed on downstream tasks compared to mBERT, potentially due to "catastrophic overtraining."

5. Low-Resource Performance:

   • For Quechua (qu), the non-embedding strategy with a custom tokenizer significantly outperformed full tuning, validating the approach for extremely low-resource scenarios.

6. Comparison with LoRA:

   • The proposed method consistently outperformed LoRA (Low-Rank Adaptation) across all tasks, suggesting that the low-rank constraint may limit the model's capacity to learn complex tasks in these specific settings.

Significance and Claims

The paper claims that full model finetuning is not always the optimal strategy for adapting PMLMs to low-resource languages. Instead, a modular approach that combines custom tokenization with selective parameter training (freezing embeddings) offers a simple, parameter-efficient, and effective alternative.

Key takeaways emphasized by the authors include:

• Regularization: Freezing embeddings acts as a powerful regularizer, mitigating overfitting in data-scarce environments.
• Tokenization is Critical: The choice of a custom tokenizer provides a more significant performance boost than the specific method used to initialize the embeddings.
• Efficiency: The approach reduces trainable parameters by roughly half, leading to faster convergence, lower VRAM usage, and reduced inference latency without sacrificing downstream task performance.
• Transferability: Multilingual pretrained models remain valuable even for languages not well-represented in their pretraining data, provided an appropriate adaptation strategy is used.

The authors conclude that this method provides a robust baseline for low-resource language modeling, avoiding the computational costs and overfitting risks associated with full model tuning.