💬 NLP

Expanding the Lexicon of Ge'ez Based African Languages: A Comparative Study of Amharic and Tigrinya

This paper introduces VEXMLM, a vocabulary-extended variant of XLM-R tailored for Ge'ez-script languages like Amharic and Tigrinya, which significantly improves performance on downstream tasks such as question answering, sentiment analysis, and named entity recognition by reducing out-of-vocabulary rates and excessive subword fragmentation through a specialized two-stage training strategy that also benefits 17 other low-resource African languages.

Original authors: Hailay Kidu Teklehaymanot, Debela Desalegn Yadeta, Wolfgang Nejdl

Published 2026-07-17

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Hailay Kidu Teklehaymanot, Debela Desalegn Yadeta, Wolfgang Nejdl

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet as a giant, bustling library where books are written in thousands of different languages. For a long time, the librarians (the smart computer programs we call Artificial Intelligence) only knew how to read books written in the Latin alphabet—the kind used for English, Spanish, and French. They had a massive dictionary for these languages, but when they tried to read books written in other scripts, like the Ge'ez script used in Ethiopia and Eritrea, they got confused. They would chop words into tiny, meaningless pieces or just throw up their hands and say, "I don't know this word!" This is a problem because it leaves millions of people out of the digital conversation, unable to use the best tools for understanding text, finding answers, or recognizing important names.

To fix this, researchers have been trying to teach these AI models new words. But it's tricky. If you just force new words into an old dictionary without teaching the AI how they fit together, the AI gets lost. This paper dives into that specific challenge: how to upgrade a super-smart AI so it can finally read and understand Amharic and Tigrinya, two major languages written in the beautiful Ge'ez script, without breaking the rest of its brain.

The Problem: The "Glue" That Doesn't Stick

Think of a modern AI language model like a master chef who can cook amazing meals in 100 different languages. But this chef has a very specific rule: they only use a pre-packaged set of ingredients (called "tokens") that were mostly designed for Latin-script languages. When the chef tries to cook a dish in Amharic or Tigrinya, the ingredients don't fit. The script is different; it's a "syllabic abugida," meaning each symbol represents a consonant and a vowel together, rather than just a single letter.

Because the chef's ingredient list is wrong for these languages, they end up chopping the words into tiny, useless crumbs. A single word might get split into five or six pieces, and many words don't exist in the chef's pantry at all. When the AI encounters a word it doesn't know, it just labels it "UNK" (unknown), throwing away all the meaning. This makes the AI terrible at tasks like finding a person's name in a news article or figuring out if a tweet is happy or sad.

The Solution: VEXMLM, the Custom Tailor

The researchers introduced a new approach called VEXMLM. Imagine VEXMLM as a master tailor who takes the chef's existing, high-quality suit (the AI model) and custom-fits it for a new body type. They didn't just throw new ingredients into the pantry; they did it with a specific, careful recipe.

First, they built a brand-new, custom dictionary specifically for Amharic and Tigrinya. They took 30,000 new subword pieces (chunks of words) that perfectly match the Ge'ez script and added them to the AI's vocabulary. This is like giving the chef a new set of specialized knives and spices that actually fit the ingredients they are trying to cook.

But here is the clever part: you can't just paste new ingredients into a recipe and expect them to taste right immediately. The new words need to know how to behave. The researchers used a "mean initialization" trick. Instead of randomly guessing what the new words should mean, they calculated the average "flavor" of all the existing words in the AI's brain and gave the new words that average flavor to start with. This ensures the new words fit in smoothly without shocking the system.

Finally, they didn't just stop at adding the words. They let the AI "study" for a while using large collections of Amharic and Tigrinya text. This is the "continued pretraining" phase. It's like letting the chef practice cooking with the new ingredients until they master the taste. After this study session, they tested the AI on specific tasks like answering questions, recognizing names, and detecting sentiment.

The Results: A Big Leap Forward

The results were impressive. When tested on Amharic and Tigrinya, the new VEXMLM model was a clear winner.

Question Answering: When asked to find answers in a text, the old model (XLM-R) got the exact match right 66% of the time. VEXMLM jumped that up to 87%. That's a huge improvement, meaning the AI is much better at understanding the full context of a sentence.
Sentiment Analysis: For figuring out if a message is positive or negative, VEXMLM reached 80% accuracy, beating the old model's 77% and crushing a massive competitor model (Glot500) which only got 46%.
Recognizing Names (NER): This is where the "unknown word" problem usually hurts the most. The old model struggled with words it didn't know, getting them right only 81.4% of the time. VEXMLM boosted that to 94.3%.

Interestingly, the researchers found that while the new vocabulary was built specifically for Amharic and Tigrinya, the benefits spilled over to other African languages too. Even languages that don't use the Ge'ez script saw improvements in recognizing out-of-vocabulary words. The authors suggest this happened because the "study session" (continued pretraining) helped the AI get better at handling complex word structures in general, not just the specific new words they added.

What This Means

The paper shows that you don't need to build a giant, expensive AI from scratch to help low-resource languages. Instead, you can take a powerful existing model, give it a custom vocabulary tailored to its specific script, and let it practice with real text. The study proves that this targeted approach works better than just making the model bigger or hoping it learns on its own.

However, the researchers are careful to note that this isn't a magic wand for every problem. The improvements were most dramatic for the languages they specifically trained on (Amharic and Tigrinya). For other languages, the gains were a helpful side effect of the training process, not a direct result of the new vocabulary. They also point out that their model still has limits, like a maximum sentence length it can read at once, which might make it struggle with very long documents.

In the end, VEXMLM is a proof of concept: with the right tools and a little bit of focused practice, we can make AI much more inclusive, ensuring that speakers of Ge'ez-script languages aren't left behind in the digital age.

Technical Summary: Expanding the Lexicon of Ge'ez-Based African Languages

Problem Statement
Multilingual pre-trained language models (PLMs), such as XLM-R, exhibit degraded performance on low-resource, non-Latin script languages. This issue is particularly acute for Ge'ez-script languages like Amharic and Tigrinya. Standard tokenization strategies, trained predominantly on Latin-script corpora, result in excessive subword fragmentation and high out-of-vocabulary (OOV) rates for morphosyllabic scripts. Consequently, tokens are often mapped to generic [UNK] symbols, discarding lexical information and severely impacting token-sensitive tasks such as Named Entity Recognition (NER), Sentiment Analysis (SA), and Question Answering (QA). While existing broad-coverage models like Glot500 offer scale, they do not optimize tokenization for specific script families, leaving a representational gap for Ge'ez languages.

Methodology: VEXMLM
The authors propose VEXMLM (Vocabulary-Extended XLM), a framework designed to adapt XLM-R to Ge'ez-script languages through a systematic four-component approach:

Language-Specific Tokenization and Vocabulary Expansion:
- Independent SentencePiece tokenizers are trained on curated monolingual corpora for Amharic (32,000 subwords) and Tigrinya (50,000 subwords).
- A new vocabulary is constructed by retaining only tokens from these tokenizers that are absent from XLM-R's original vocabulary ( $V_s$ ).
- This results in the addition of 30,000 net new Ge'ez-derived subword tokens, expanding the total vocabulary from 250K to 280K tokens while preserving backward compatibility with all original XLM-R representations.
Mean-Based Embedding Initialization:
- To integrate new tokens without destabilizing the pre-trained embedding space, the authors employ a mean initialization strategy.
- New token embeddings are initialized to the geometric centroid (mean) of the source embeddings from the original vocabulary. This avoids the biases of zero-initialization and the instability of random initialization, positioning new tokens at the center of the existing embedding distribution.
Two-Stage Training Procedure:
- Stage 1 (Continued Pretraining): The model undergoes continued Masked Language Modeling (MLM) on large, unlabeled monolingual corpora for Amharic and Tigrinya. All parameters, including the newly initialized embeddings, are updated. Low-resource data is upsampled to address corpus imbalance.
- Stage 2 (Supervised Fine-Tuning): The model is fine-tuned on downstream tasks (NER, SA, QA). To mitigate overfitting in resource-constrained settings, only the task-specific classification heads and the embedding layer are updated, while the transformer body is partially frozen.
Evaluation Scope:
- The model is evaluated on 19 low-resource African languages, with a primary focus on Amharic and Tigrinya.
- Evaluation includes intrinsic metrics (tokenization parity, fertility, compression, OOV accuracy) and extrinsic metrics (task performance on NER, SA, and QA).

Key Contributions
The paper identifies three principal contributions:

Ge'ez-Specific Vocabulary Expansion: A principled approach to augmenting vocabulary with 30,000 Ge'ez-derived subwords and initializing their embeddings via the mean of source embeddings to ensure compatibility with the pre-trained space.
Cross-Lingual Generalization via Two-Stage Training: A demonstration that continued MLM pretraining on Ge'ez-script monolingual corpora, followed by fine-tuning, enables strong transfer to 17 additional African languages that were not targets of the vocabulary expansion.
Comprehensive Evaluation: A thorough characterization of the model using both tokenization efficiency metrics and downstream task performance across 19 languages.

Results
Experimental results indicate that VEXMLM substantially outperforms the baseline XLM-R and the broad-coverage Glot500 on specific tasks:

Question Answering (QA): On Amharic/Tigrinya QA, VEXMLM achieves an Exact Match (EM) of 87.0 and F1 of 90.0, compared to XLM-R (66.0 EM / 78.0 F1) and Glot500 (74.0 EM / 78.0 F1).
Sentiment Analysis (SA): VEXMLM reaches 80.0% accuracy, outperforming XLM-R (77.0%) and significantly surpassing Glot500 (46.0%).
Named Entity Recognition (NER): VEXMLM improves OOV-token entity accuracy from 81.4% to 94.3% on average across 11 languages where OOV analysis was possible. Notably, while the vocabulary expansion targets Ge'ez scripts, OOV accuracy improved in non-Ge'ez languages (e.g., Swahili +15.0 points, Kinyarwanda +8.5 points), attributed to the continued pretraining phase.
Ablation Studies: The study confirms that continued pretraining is the most critical component, providing the largest performance gain (+7.1 points in OOV accuracy). Mean initialization also proved superior to random initialization, enabling faster and more stable convergence.

Significance and Claims
The paper claims that VEXMLM successfully narrows the representational gap for underserved Ge'ez-script communities by addressing the specific technical bottlenecks of subword tokenization and vocabulary coverage. The authors emphasize that their approach offers a "middle ground": targeted script-specific coverage with modest parameter overhead (301M parameters vs. Glot500's 395M).

The work highlights that while vocabulary expansion is script-specific, the benefits of the two-stage training pipeline (specifically continued pretraining) generalize to other low-resource African languages, improving the model's ability to represent morphologically complex tokens even in languages without the expanded vocabulary. The authors position this as a step toward more equitable multilingual NLP, though they acknowledge limitations regarding the scope of vocabulary expansion (limited to Amharic and Tigrinya), data constraints, and the need for future controlled comparisons with other African-specific models like AfroXLMR and AfriBERTa.

The Problem: The "Glue" That Doesn't Stick

The Solution: VEXMLM, the Custom Tailor

The Results: A Big Leap Forward

What This Means

Technical Summary: Expanding the Lexicon of Ge'ez-Based African Languages

More like this