💬 NLP

Evaluating Large Language Models for Hausa and Fongbe Machine Translation: Benchmarks, Failures, and Metric Reliability

This study evaluates the translation capabilities of four large language models for English-to-Hausa and English-to-Fongbe, revealing significant performance disparities between the two languages, the unreliability of standard automatic metrics in correlating with human judgment, and the necessity of large-scale evaluation and multi-metric approaches for low-resource African languages.

Original authors: Mahounan Pericles Adjovi, Roald Eiselen, Prasenjit Mitra

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mahounan Pericles Adjovi, Roald Eiselen, Prasenjit Mitra

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to translate a story from English into two very different African languages: Hausa (spoken by millions in Nigeria and Niger) and Fongbe (spoken by a smaller group in Benin). You have four powerful "robot translators" (Large Language Models like GPT-4, Claude, and Gemini) ready to do the job.

This paper is like a rigorous taste test to see which robot does the best job and, more importantly, whether the "scorecards" we use to grade them (automatic metrics) actually match what real human speakers think.

Here is the breakdown of their findings, using simple analogies:

1. The "Robot Race": Who Won?

The researchers asked the robots to translate thousands of sentences. The results were a tale of two very different worlds:

Hausa (The "Good Enough" Runner): The robots did a decent job here. The translations were understandable and flowed well. It was like a runner finishing a race with a respectable time.
- The Winner: GPT-4o was the favorite among human judges, even though the computer scorecards gave the win to Claude.
Fongbe (The "Stumbling" Runner): The robots struggled badly here. The translations were often nonsensical or broken. It was like a runner tripping over their own shoelaces.
- The Winner: Gemini was the least bad option, but even it only scored a "poor" grade from humans.
The Big Gap: Hausa translations were roughly three times better than Fongbe translations. This wasn't because Fongbe is a "harder" language, but because the robots had seen way more training data for Hausa than for Fongbe.

Key Takeaway: Just because a robot is good at one African language doesn't mean it will be good at another. You can't assume a "one-size-fits-all" AI works everywhere.

2. The "Scorecard" Problem: Do the Computers Lie?

This is the most surprising part of the paper. The researchers compared the robots' performance against two things:

Human Judges: Real native speakers rating the translations.
Automatic Metrics: Computer algorithms (like BLEU, chrF++, BERTScore) that try to grade translations without a human looking at them.

The Mismatch:

For Fongbe: The computer scorecards and the humans agreed. They both said, "Gemini is the best."
For Hausa: The computer scorecards and the humans disagreed completely.
- The computers (specifically BLEU and chrF++) said: "Claude is the winner!"
- The humans said: "No, GPT-4o is the winner; Claude sounds robotic."

The "Broken Ruler" Analogy:
Imagine you are measuring the height of two people.

For Fongbe, your ruler works perfectly.
For Hausa, your ruler is bent. It tells you the shorter person is taller because it's measuring the wrong thing (like measuring how many letters are in a word instead of how well the sentence sounds).

3. The "Magic Mirror" That Reflects Nothing (Neural Metrics)

The paper tested a specific type of computer metric called BERTScore, which uses a "magic mirror" (an embedding model) to see if two sentences mean the same thing.

The Glitch: For both Hausa and Fongbe, this mirror was broken. It looked at two completely different sentences and said, "These are 99% identical!"
The Result: Because the mirror couldn't tell the difference between a good translation and a bad one, the scores were all the same (high numbers for everyone). It was like a judge giving every contestant a gold medal because they couldn't tell who was actually singing.
The Lesson: You cannot trust these "smart" metrics for these languages until we fix the mirror.

4. The "Sample Size" Trap

The researchers tried testing the robots with small groups of sentences (500 or 1,000) and then with a huge group (10,000).

The Trap: With small groups, the results were chaotic and misleading. For example, at 1,000 sentences, the data suggested Gemini was winning Hausa. But when they tested 10,000 sentences, it turned out Claude was actually winning.
The Lesson: You need a large crowd (at least 2,500 sentences) to get a true picture. Small samples are like judging a whole movie based on a 10-second clip; you might get the wrong idea.

Summary of Recommendations

Based on this "taste test," the authors suggest:

Don't trust the computer scorecards alone. Especially for Hausa, the computers got the winner wrong. You must have real humans check the work.
Use the right ruler. If you must use a computer metric, use chrF++ (which counts character matches) rather than the "magic mirror" (BERTScore), which is currently broken for these languages.
Pick your robot carefully. If you need to translate to Hausa, use GPT-4o or Claude. If you need to translate to Fongbe, use Gemini, but be prepared for poor quality.
Don't use Fongbe translations for anything serious yet. The quality is too low; it would be like trying to build a house with wet sand.
Hausa is ready for "data augmentation." The translations are good enough to be used as extra training data, provided you filter out the bad ones first.

In short: AI is getting better at translating African languages, but it's not there yet. The tools we use to measure its success are often broken, and we need real humans to keep the robots honest.

Technical Summary: Evaluating Large Language Models for Hausa and Fongbe Machine Translation

Problem Statement

Machine translation (MT) for low-resource African languages is hindered by critical data scarcity. While Large Language Models (LLMs) offer a potential solution through synthetic parallel data generation, their ability to produce sufficient translation quality for typologically distinct languages remains unverified. Furthermore, the reliability of standard automatic evaluation metrics—developed primarily on high-resource European languages—has not been systematically validated for these specific language pairs.

This study addresses two specific research questions:

RQ1: What is the translation quality of current commercial and open-weight LLMs for English-to-Hausa and English-to-Fongbe, and how does this quality vary across systems and languages?
RQ2: How reliably do automatic evaluation metrics correlate with native-speaker human judgment for these languages, and which metrics should guide quality filtering?

The investigation focuses on Hausa (Afroasiatic family, 58M speakers) and Fongbe (Niger-Congo family, 2.3M speakers). These languages present distinct linguistic challenges: Hausa features agglutinative morphology and complex verbal TAM marking, while Fongbe utilizes serial verb constructions and a three-tone system that is obligatorily marked orthographically.

Methodology

Models Evaluated

The study evaluated four LLM-based translation systems at progressive scales (500 to 10,000 sentences):

Commercial APIs: GPT-4o Mini (OpenAI), Claude Sonnet 4 (Anthropic), Gemini 2.5 Flash (Google).
Open-Weight: Qwen2.5-7B (Alibaba).
All models were run with a temperature of 0.0 for deterministic outputs.

Data and Evaluation Strategy

Source Data: Parallel sentence pairs were sourced from the OPUS repository (CCMatrix for English–Hausa; NLLB v1 for English–Fongbe).
Scales: Evaluations were conducted at $n=500, 1,000, 2,500,$ and $10,000$ sentences to determine the minimum sample size required for stable system rankings.
Automatic Metrics: Five metrics were employed:
- Lexical: BLEU, chrF++, TER.
- Neural: COMET (wmt22-comet-da) and BERTScore (XLM-RoBERTa-large).
Human Evaluation: 13 native speakers (7 Fongbe, 6 Hausa) assessed translations on a 5-point scale across four dimensions: Adequacy, Fluency, Cultural Appropriateness, and Overall Quality. Due to resource constraints, $n=50$ sentences were evaluated per language by 3 evaluators each.
Statistical Analysis: Bootstrap resampling (10,000 iterations) was used to compute 95% confidence intervals for pairwise system comparisons. Spearman's $\rho$ was used to measure rank correlation between metrics and human judgments.

Key Contributions

LLM Benchmark for Hausa and Fongbe: This is the first systematic comparison of current commercial LLMs (released 2024–2025) for these specific languages with statistical significance testing at scale ( $n=10,000$ ). It reveals that performance on one low-resource African language does not predict performance on another.
Metric Reliability Analysis: The study provides a systematic validation of automatic metrics against native-speaker judgment. It identifies that metric-human correlation varies dramatically by language and exposes "embedding collapse" in neural metrics for both languages.
Practical Recommendations: The authors establish minimum sample size requirements ( $n=2,500$ ) for reliable evaluation and recommend specific metrics (chrF++) and evaluation protocols (mandatory human evaluation) for low-resource African languages.

Results

Translation Quality and Model Rankings

Cross-Language Performance Gap: Hausa translations achieved acceptable quality (Human scores 3.37–4.46/5), whereas Fongbe translations were poor (1.0–2.2/5). A consistent 3× BLEU gap existed between the best Hausa system and the best Fongbe system across all scales.
Language-Specific Model Rankings:
- Fongbe: Gemini 2.5 Flash led consistently across both automatic metrics and human evaluation.
- Hausa: GPT-4o Mini was preferred by human evaluators (4.46/5), while Claude Sonnet 4 ranked first on all automatic metrics (BLEU 15.75).
- Open-Source: Qwen2.5-7B significantly underperformed all commercial models.

Metric Reliability and Correlation

Metric-Human Correlation:
- Fongbe: Perfect rank correlation ( $\rho=1.0$ ) between automatic metrics and human judgment.
- Hausa: Weak correlation ( $\rho=0.5$ ). Human evaluators preferred GPT-4o, while all automatic metrics ranked Claude first. This indicates that surface-level and semantic metrics may miss quality dimensions (e.g., naturalness, cultural appropriateness) prioritized by native speakers.
Neural Metric Failures (Embedding Collapse):
- BERTScore: Exhibited "embedding collapse" for both languages, with within-language cosine similarity exceeding 0.99. This rendered the metric unable to differentiate translation quality, showing uniformly high scores (0.83–0.88) regardless of actual performance.
- COMET: Showed anomalous behavior for Fongbe, ranking the worst-performing system (Qwen2.5-7B) first despite its poor lexical scores. For Hausa, COMET aligned better with lexical metrics.
Scale Stability: Rankings were unstable at $n=1,000$ (e.g., Gemini appeared to lead Hausa erroneously). Stable, statistically significant rankings required a minimum of $n=2,500$ sentences.

Reference Corpus Issues

Post-hoc analysis of the reference data revealed significant quality issues, particularly for Fongbe, including wrong-language contamination (Chinese instead of Fongbe) and encoding corruption. The average quality score of the reference corpus was 2.94/5.0.

Significance and Claims

The paper claims that "low-resource" is not a monolithic category; substantial quality gaps exist even among West African languages with millions of speakers. The study demonstrates that:

Evaluation Practices Do Not Transfer: Metrics validated on high-resource languages do not reliably transfer to typologically distinct, low-resource settings. Specifically, neural metrics like BERTScore are unreliable due to embedding collapse in models like XLM-RoBERTa, even for languages included in pre-training (Hausa).
Scale is Critical: Small sample sizes ( $<2,500$ ) produce artifact findings that reverse at scale. Reliable system comparison for low-resource MT requires larger datasets.
Model Selection is Language-Specific: No single model excels at both languages. Performance on one low-resource language does not predict performance on another, suggesting that training data representation is the dominant factor rather than a generalizable "low-resource capability."
Data Augmentation Viability:
- Hausa: LLM-generated translations are viable for data augmentation if filtered using thresholds (e.g., COMET > 0.55 or chrF++ > 35).
- Fongbe: LLM-generated translations are not viable for data augmentation, as the best system (Gemini) scored below the acceptable threshold (2.20/5), introducing more noise than signal.

The authors conclude that for morphologically complex and low-resource languages, chrF++ should be the primary automatic metric, but mandatory human evaluation is essential for final quality assessment, particularly when automatic metrics diverge from human judgment.