LGSE: Lexically Grounded Subword Embedding Initialization for Low-Resource Language Adaptation

1. Problem Statement

Pretrained Multilingual Language Models (PLMs) like XLM-R and mBERT struggle with low-resource, morphologically rich languages (e.g., Amharic and Tigrinya). The core issues identified are:

• Arbitrary Segmentation: Standard tokenizers (like Byte Pair Encoding - BPE) rely on character co-occurrence frequencies rather than linguistic structure. This leads to the fragmentation of meaningful words into semantically meaningless subword units.
• Loss of Morphological Integrity: In agglutinative or templatic languages, BPE often breaks complete lexical units (e.g., nouns) into noise, destroying morphemic boundaries.
• Poor Embedding Initialization: When expanding vocabularies for new tokens, existing methods often use random vectors or simple averaging of arbitrary subwords. This results in embeddings that lack semantic grounding and fail to capture the deep morphological structure of the target language.
• Alignment Disruption: Simply adding new tokens and retraining often disrupts the alignment with the original pretrained embedding space, leading to suboptimal performance.

2. Methodology: The LGSE Framework

The authors propose Lexically Grounded Subword Embedding Initialization (LGSE), a framework that integrates morphologically aware tokenization with linguistically informed embedding initialization.

A. Morphologically Aware Tokenization

Instead of standard BPE, the authors utilize a hybrid tokenization strategy (based on their previous work, MoVoC):

1. Morpheme Segmentation: Words are first segmented into linguistically valid morphemes using supervised morphological analysis.
2. Subword Refinement: BPE is applied within each morpheme, but merges are prevented across morpheme boundaries.
3. Hybrid Vocabulary: The final vocabulary combines high-frequency morphemes with subword units, controlled by a ratio hyperparameter $r$.

B. Lexically Grounded Embedding Initialization

For new tokens introduced during vocabulary expansion, LGSE constructs embeddings based on their morphological composition rather than random initialization:

1. Morpheme Decomposition: A new token $t$ is decomposed into morphemes $M(t) = [m_1, m_2, ..., m_k]$.
2. FastText Composition: Each morpheme is represented by averaging the character $n$-gram embeddings derived from FastText.
   • $m_j = \frac{1}{|G_j|} \sum_{g \in G_j} f_g$
3. Token Embedding Construction: The initial token embedding is the average of its constituent morpheme embeddings.
   • $e_t = \frac{1}{k} \sum_{j=1}^{k} m_j$
4. Space Alignment: A learned linear projection $W$ aligns the FastText embedding space with the PLM's embedding space ($e_{aligned} = W e_t$).
5. Regularization: During Language-Adaptive Pretraining (LAPT), a regularization term ($L_{reg} = \lambda \|e_{new} - \mu\|^2$) is applied to penalize large deviations from the initialized values, preserving alignment with the original pretrained space while allowing adaptation.

C. Experimental Setup

• Base Model: XLM-R (125M parameters).
• Languages: Amharic and Tigrinya (Ge'ez script).
• Pretraining: Language-Adaptive Pretraining (LAPT) on monolingual corpora (CC100 for Amharic; custom data for Tigrinya) using Masked Language Modeling (MLM).
• Baselines:
  • XLM-R (Off-the-shelf).
  • XLM-R + LAPT (Standard adaptation).
  • XLM-R + Random Init + LAPT.
  • XLM-R + FOCUS (Subword refinement) + LAPT.
  • XLM-R + LGSE + LAPT (Proposed).

3. Key Contributions

1. Diagnosis of Subword Limitations: The paper empirically demonstrates that standard subword embeddings in PLMs fail to capture the morphological structure of low-resource languages, leading to fragmented and semantically weak representations.
2. LGSE Framework: Introduction of a novel initialization strategy that leverages morpheme-aware segmentation and FastText-based compositional embeddings to create semantically coherent vectors for new tokens.
3. New Benchmark Dataset: Creation of the first human-annotated benchmark for Educational Quality Classification in Amharic and Tigrinya (2,500 samples), addressing a gap in low-resource educational AI evaluation.
4. Comprehensive Evaluation: Rigorous testing across three downstream tasks (Question Answering, Named Entity Recognition, Text Classification) showing consistent improvements over strong baselines.

4. Results

The experiments were conducted on Amharic and Tigrinya across three tasks. The LGSE + LAPT model consistently outperformed all baselines.

• Question Answering (QA):
  • LGSE achieved an average F1 score of 78.25, compared to 76.65 for FOCUS+LAPT and 66.35 for the off-the-shelf model.
• Named Entity Recognition (NER):
  • LGSE achieved an average F1 score of 79.20, significantly outperforming FOCUS+LAPT (77.80) and Random+LAPT (72.15).
• Text Classification (TC):
  • LGSE achieved an average accuracy of 76.50, surpassing FOCUS+LAPT (74.45).

Efficiency Metrics:

• Token Fertility (TF): LGSE significantly reduced the number of tokens per word. For example, a Tigrinya sentence was tokenized into 6 tokens by LGSE versus 21 tokens by standard BPE.
• Inference Latency: Reduced sequence lengths led to lower inference latency, demonstrating computational efficiency.

5. Significance and Impact

• Linguistic Grounding: The paper proves that respecting linguistic boundaries (morphemes) during both tokenization and embedding initialization is critical for low-resource languages. It moves beyond statistical frequency to structural understanding.
• Cross-Lingual Transfer: The method effectively bridges the gap for languages with no pretraining data (e.g., Tigrinya) by leveraging morphological overlap and compositional semantics from related languages (e.g., Amharic) and FastText resources.
• Resource Equity: By improving performance on underrepresented languages without requiring massive new pretraining corpora, LGSE offers a scalable path to more equitable NLP systems.
• Educational AI: The release of the educational quality dataset provides a foundation for developing AI tools specifically for African educational contexts.

In conclusion, LGSE demonstrates that morphology-aware initialization is a superior alternative to random or arbitrary subword initialization, enabling PLMs to better understand and process the complex grammatical structures of low-resource languages.