ANGOFA: Leveraging OFA Embedding Initialization and Synthetic Data for Angolan Language Model

Technical Summary: ANGOFA: Leveraging OFA Embedding Initialization and Synthetic Data for Angolan Language Models

Problem Statement
Despite recent advancements in pre-trained language models (PLMs) that facilitate cross-lingual knowledge transfer, very-low-resource languages remain significantly underrepresented. Specifically, Angolan languages (part of the Niger-Congo/Bantu family) have been largely overlooked by both general multilingual models and existing African-centric initiatives. The scarcity of native literature, broadcast media, and digital corpora for these languages creates a barrier to developing effective natural language processing (NLP) tools. Furthermore, standard approaches to adapting PLMs to new languages often rely on random initialization for new vocabulary tokens, which fails to leverage existing lexical knowledge and can hinder efficient training.

Methodology
The authors introduce ANGOFA, a set of multilingual PLMs tailored for five Angolan languages: Chokwe, Kimbundu, Kikongo, Luba-Kasai, and Umbundu. The development process employs a Multilingual Adaptive Fine-tuning (MAFT) strategy rather than training from scratch, chosen for its resource efficiency. The methodology integrates three key technical components:

1. Vocabulary Expansion: The base model's vocabulary is expanded to include tokens for the target Angolan languages to address Out-of-Vocabulary (OOV) issues.
2. OFA Embedding Initialization: Instead of random initialization for new subword embeddings, the authors utilize OFA (Optimal Factorized Adaptation). This approach factorizes the source model's embedding matrix and leverages external multilingual embeddings (specifically ColexNet+) to initialize new subwords. This ensures that embeddings for new subwords are expressed as weighted combinations of source subword embeddings based on cross-lingual similarity, preserving representation continuity.
3. Synthetic Data Augmentation: To overcome data scarcity, the authors generate synthetic monolingual data using machine translation (specifically the NLLB-600M model) from an English news commentary dataset. This synthetic data is combined with existing monolingual corpora to create a larger training set.

The authors compare two primary model variants:

• ANGXLM-R: A model fine-tuned via MAFT with random embedding initialization.
• ANGOFA: A model fine-tuned via MAFT with OFA embedding initialization.

Both variants are trained in two configurations: one using only monolingual data and another incorporating the synthetic data.

Key Contributions

• First PLMs for Angolan Languages: The paper presents the first set of multilingual PLMs specifically tailored for five major Angolan languages.
• Empirical Validation of OFA in Low-Resource Settings: The study demonstrates that informed embedding initialization (OFA) significantly outperforms random initialization when adapting models to low-resource languages.
• Impact of Synthetic Data: The research quantifies the performance gains achieved by augmenting limited monolingual corpora with machine-translated synthetic data.
• Benchmarking: The models are evaluated on SIB-200, a text classification dataset covering over 200 African languages, providing a standardized benchmark for Angolan language performance.

Results
The models were evaluated on the SIB-200 text classification task using a weighted F1 score. The results indicate:

• Region-Specific Adaptation: Models adapted specifically to Angolan languages via MAFT outperformed general pre-trained models (XLM-R, Serengeti, Glot-500) and broader African models (AfroXLMR-base) that included fewer Angolan languages.
• Synthetic Data Efficacy: Incorporating synthetic data improved the performance of the MAFT model (ANGXLM-R) by +5.5 points over the version trained without synthetic data.
• OFA Superiority: The ANGOFA model (OFA initialization + synthetic data) achieved the best overall performance.
  • It outperformed the baseline AfroXLMR-base by 12.3 points.
  • It outperformed the OFA baseline (trained on 500 languages without specific Angolan synthetic data) by 3.8 points.
  • It achieved a total improvement of +16.6 points over the original XLM-R baseline.
• Data vs. Initialization: Interestingly, the authors note that without synthetic data, ANGOFA performed worse than the OFA baseline trained on a larger 500-language corpus. However, with synthetic data, the specific adaptation to Angolan languages yielded superior results, suggesting that targeted data volume can outweigh the benefits of broader, less specific pre-training in this context.

Significance and Claims
The paper claims that the combination of informed embedding initialization (OFA) and synthetic data augmentation is a critical strategy for developing effective PLMs for very-low-resource languages. The authors argue that region-specific models, when properly adapted with these techniques, can outperform massive, general-purpose models that lack deep coverage of specific language families.

The authors maintain a modest tone regarding the findings, noting that while OFA generally provides superior results, the specific factors driving the performance of ANGOFA over OFA in certain languages (such as Luba-Kasai) raise questions about the interplay between dataset size and embedding initialization strategies. These nuances are identified as areas for future investigation. The work aims to contribute to the broader goal of reducing the digital divide for African languages by providing accessible, high-performance models for Angolan NLP tasks.