💬 NLP

LLM Probe: Evaluating LLMs for Low-Resource Languages

This paper introduces LLM Probe, a lexicon-based evaluation framework and accompanying benchmark dataset for low-resource languages, which reveals distinct performance differences between causal and sequence-to-sequence models across tasks like lexical alignment, morphosyntactic analysis, and translation to advocate for linguistically grounded assessment in multilingual AI development.

Original authors: Hailay Kidu Teklehaymanot, Gebrearegawi Gebremariam, Wolfgang Nejdl

Published 2026-04-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hailay Kidu Teklehaymanot, Gebrearegawi Gebremariam, Wolfgang Nejdl

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot that can speak thousands of languages. It's like a polyglot genius who can write poetry, solve math problems, and tell jokes in English, Spanish, and Mandarin. But, there's a catch: this robot has mostly learned from books written in those popular languages. If you ask it about a language spoken by a few million people in a specific region—like Tigrinya (spoken in Eritrea and Ethiopia)—it might stumble, guess, or just make things up.

This paper is about building a specialized "report card" to test exactly how well these robots handle languages they haven't studied enough. The authors call this test "LLM Probe."

Here is the breakdown of the paper using simple analogies:

1. The Problem: The "Rich vs. Poor" Language Gap

Think of the world's languages like neighborhoods.

High-Resource Languages (like English): These are like bustling cities with massive libraries, millions of teachers, and endless textbooks. The AI robots have studied here for years, so they are experts.
Low-Resource Languages (like Tigrinya): These are like small, vibrant villages with a rich oral history but very few written books. The AI robots have barely visited these villages.

Because Tigrinya is a "morphologically rich" language, it's like a Lego set with complex, interlocking pieces. In English, you might just add an "s" to make a word plural. In Tigrinya, you have to twist, turn, and change the internal vowels of the word to show who is doing the action, when, and to whom. The robots often get confused by these complex Lego structures because they haven't seen enough examples.

2. The Solution: The "LLM Probe" Toolkit

Instead of just asking the robot to "translate this sentence" and hoping for the best, the authors built a four-part diagnostic toolkit (the LLM Probe) to check the robot's brain in specific ways:

Lexical Alignment (The "Word Match" Game):
- Analogy: Imagine a game of "Memory" where you have to match an English word card to its Tigrinya twin.
- The Test: Can the robot correctly pair "cat" with the Tigrinya word for cat, without mixing them up?
Part-of-Speech Tagging (The "Job Title" Check):
- Analogy: You hand the robot a list of words and ask, "Is this word a noun (a thing), a verb (an action), or an adjective (a description)?"
- The Test: Can the robot understand the role a word plays in a sentence, even if the sentence structure is weird?
Morphosyntactic Probing (The "Grammar Puzzle"):
- Analogy: This is like checking if the robot knows the rules of a complex board game. Does it know that if the subject is "he," the verb must change shape?
- The Test: Can the robot handle the gender (male/female) and number (singular/plural) changes that Tigrinya requires?
Translation Fidelity (The "Accuracy Check"):
- Analogy: The robot translates a sentence, and a human expert compares it to the perfect translation.
- The Test: Did the robot get the meaning right, or did it just sound fluent but say something wrong?

3. The Data: Building the "Gold Standard"

To run these tests, the authors couldn't just use Google Translate. They had to build their own manual benchmark dataset.

They gathered 7,234 pairs of English and Tigrinya phrases.
They hired native speakers (linguists) to manually label every single word with its grammar rules, gender, and meaning.
Why? To ensure the "answer key" is 100% correct. If the answer key is wrong, you can't tell if the robot is smart or stupid.

4. The Results: What the Robots Got Right (and Wrong)

The authors tested several different robot models (some are "Causal" models, which predict the next word like a text message; others are "Sequence-to-Sequence" models, which are better at translating whole sentences).

The Winners: The Sequence-to-Sequence models (like mT5 and ByT5) were the best at the complex grammar puzzles and translation. They are like specialized translators who understand the whole sentence structure.
The Losers (in some areas): The Causal models (like Gemma or Falcon) were great at matching individual words but struggled to translate whole sentences accurately. They are like word-matching experts who sometimes lose the plot of the story.
The Surprise: Even the best robots still made mistakes with the complex "Lego" grammar of Tigrinya. They often confused nouns with verbs or got the gender wrong.

5. Why This Matters

The paper concludes that we can't just assume AI is good at everything. If we want AI to help people in Ethiopia or Eritrea, we need to:

Stop guessing: We need tests like LLM Probe to see exactly where the robots fail.
Build better tools: We need more "Lego instruction manuals" (dictionaries and grammar tools) for low-resource languages.
Listen to locals: The best way to build these tools is to work with native speakers, not just computer scientists.

In a nutshell: This paper is a call to action. It says, "Let's stop treating all languages the same. Let's build better, fairer tests to help our AI robots learn the complex, beautiful languages of the world that have been left behind."

1. Problem Statement

Despite rapid advancements in Large Language Models (LLMs), their linguistic capabilities in low-resource and morphologically rich languages remain poorly understood. Current evaluation frameworks often rely on large-scale corpora that may already exist in pre-training data, leading to inflated performance metrics that reflect memorization rather than genuine linguistic understanding.

The Gap: There is a lack of standardized, linguistically grounded evaluation frameworks for low-resource languages.
The Specific Challenge: Languages like Tigrinya (a Semitic language spoken in Eritrea and Ethiopia) present unique difficulties due to:
- Rich Morphology: Complex root-and-pattern systems, verbal inflection, and gender/number agreement.
- Limited Digital Resources: Scarcity of annotated corpora, tokenizers, and NLP tools.
- Typological Features: Subject-Object-Verb (SOV) word order and postpositions.
Consequence: Without reliable benchmarks, it is difficult to assess how well LLMs generalize across diverse typological features or identify specific weaknesses in their competence.

2. Methodology: The LLM Probe Framework

The authors propose LLM Probe, a lexicon-based assessment framework designed to systematically evaluate LLMs in low-resource settings. Instead of relying on massive, unstructured corpora, the framework uses a manually curated, bidirectional lexicon enriched with linguistic annotations.

A. Dataset Construction

Source Material: Derived from the Swansea University Tigrinya–English dictionary, printed dictionaries, and community glossaries.
Process:
1. Digitization: Custom OCR pipeline followed by manual correction and orthographic normalization.
2. Annotation: Native-speaking linguists annotated entries with Part-of-Speech (POS) tags, morphosyntactic features (gender, number, agreement), and semantic roles.
3. Scale: The final dataset contains 7,234 annotated phrase pairs (7,068 unique English, 6,073 unique Tigrinya).
4. Quality Control: High Inter-Annotator Agreement (IAA) was achieved (Cohen's $\kappa$ ranging from 0.84 to 0.91), ensuring the reliability of the "gold standard."
Dataset Features: Includes multi-word expressions, polysemy, and bidirectional alignment (English $\leftrightarrow$ Tigrinya).

B. Evaluation Tasks

The framework evaluates models across four distinct linguistic dimensions:

Lexical Alignment: Testing word-to-word correspondence accuracy between English and Tigrinya.
Part-of-Speech (POS) Tagging: Assessing syntactic awareness by requiring models to assign grammatical categories to Tigrinya tokens.
Morphosyntactic Probing: Testing understanding of grammatical features (gender, number, agreement) and noun classes.
Translation Fidelity: Assessing semantic accuracy by comparing model outputs against human-curated reference translations (using BLEU and token-level overlap).

C. Experimental Setup

Models Evaluated: A diverse set of open-source LLMs representing two architectural paradigms:
- Causal Language Models: Falcon-10B, Gemma-2B/7B, Mistral-7B, Qwen-7B.
- Sequence-to-Sequence (Seq2Seq) Models: mT5-base/large, ByT5.
Metrics: Accuracy (for classification tasks), BLEU scores, Token-level overlap, and Confusion Matrices.
Environment: Inference conducted on high-performance GPU clusters with deterministic decoding (greedy sampling, temperature 0.0).

3. Key Contributions

LLM Probe Framework: A novel, modular, lexicon-based evaluation methodology that allows for fine-grained probing of linguistic phenomena, reducing the risk of data contamination from pre-training sets.
Tigrinya Benchmark Dataset: The creation of a high-quality, bidirectional English-Tigrinya lexicon with rich morphosyntactic annotations, addressing a critical resource gap for this underrepresented language.
Architectural Comparison: A systematic comparison of Causal vs. Seq2Seq architectures in low-resource, morphologically complex environments.
Open Source Release: The authors released both the framework and the dataset to promote reproducible research and inclusive NLP development.

4. Results

The evaluation revealed significant performance disparities based on model architecture and task type:

Sequence-to-Sequence (Seq2Seq) Superiority:
- Models like ByT5 and mT5 variants consistently outperformed causal models in Morphosyntactic Probing and Translation Fidelity (BLEU scores).
- Example: ByT5 achieved a BLEU score of 24.5, while causal models (e.g., Falcon-10B) scored around 21.8.
- ByT5 also demonstrated the highest accuracy in Lexical Alignment (1.0000), whereas mT5 struggled with this specific task (0.0059).
Causal Model Performance:
- Causal models (Gemma, Mistral, Qwen) showed strong performance in Lexical Alignment (often achieving 1.0000 accuracy) but exhibited weaker translation fidelity and morphosyntactic understanding compared to Seq2Seq models.
- They produced valid predictions but often failed to adhere to strict output formats or capture deep grammatical structures.
Error Analysis:
- Confusion matrices indicated that models frequently confused Nouns with Verbs and Adjectives, suggesting difficulty in distinguishing linguistically related categories in morphologically rich contexts where cues overlap.
- Semantic accuracy was often high even when format accuracy was low, indicating models understand the meaning but struggle with the specific syntactic constraints of the task.

5. Significance and Future Directions

Linguistic Grounding: The study emphasizes that surface-level metrics (like BLEU) can obscure deep grammatical deficiencies. A multi-dimensional, linguistically grounded approach is necessary to truly understand LLM limitations.
Inclusivity: The work highlights the need for native speaker expertise in dataset construction and evaluation to ensure cultural and linguistic validity.
Scalability: The framework serves as a template for evaluating other underrepresented languages, particularly those with complex morphology or non-Latin scripts.
Future Work: The authors plan to expand the dataset to include domain-specific vocabulary and dialectal variations, integrate automated annotation tools as they become available, and conduct deeper error analyses to guide model refinement.

Conclusion:
LLM Probe demonstrates that while current LLMs show promise in low-resource settings, their performance is highly task- and architecture-dependent. The framework provides a rigorous, reproducible method to move beyond "black box" evaluations, fostering the development of more equitable and linguistically aware multilingual technologies.