Technical Summary: Nsanku – Evaluating Zero-Shot Translation Performance of LLMs for Ghanaian Languages

Problem Statement

Despite the rapid advancement of Large Language Models (LLMs) in multilingual machine translation (MT) for well-resourced languages, their capabilities for low-resource African languages remain poorly understood and largely unevaluated. Ghana, home to over 80 documented languages, faces a critical gap: citizens often cannot access digital health, government, or educational services in their native tongues because translation tools are either non-existent or unverified. Existing multilingual benchmarks (e.g., FLORES-200, XTREME) cover at most one Ghanaian language (Twi), leaving the vast majority of the country's linguistic landscape without systematic evaluation. Furthermore, prior research has largely focused on fine-tuned models requiring labeled data, leaving the "zero-shot" capabilities of frontier LLMs—where models translate without explicit training on the specific language pair—unexplored at scale.

Methodology

The paper introduces Nsanku, a systematic benchmark designed to evaluate the zero-shot MT performance of 19 LLMs across 43 Ghanaian languages paired with English. The evaluation framework is implemented as a modular, four-stage Python pipeline:

1. Data Collection: Parallel sentence pairs were sourced from the YouVersion Bible platform, the only digital source offering sufficient breadth for 43 languages. Using a web scraper, the authors aligned Bible verses to create 300 sentence pairs per language.
2. Reproducible Sampling: A deterministic sampling step (seed 42) selected 200 sentence indices from the first language file and applied them uniformly across all 43 languages. This ensured that all 19 models were evaluated on the exact same 200 sentences per language, eliminating test set variance.
3. Model Inference: The study evaluated 19 LLMs (10 open-weight and 9 proprietary) via the NVIDIA Build API. Models included frontier systems (e.g., gemini-2.5-flash, claude-sonnet-4-5, gpt-4.1) and open-weight alternatives (e.g., kimi-k2-instruct-0905, llama-3.1-405b, deepseek-v3.1). All evaluations were conducted in a strict zero-shot setting with no in-context examples or fine-tuning.
4. Metric Computation: Performance was measured using two automatic metrics:
   • BLEU (Bilingual Evaluation Understudy): Standard n-gram precision.
   • chrF (Character n-gram F-Score): Robust to morphological variation and tokenization issues.
   • Average Score: The arithmetic mean of BLEU and chrF.
   • Consistency: Calculated as the inverse of the standard deviation of per-language scores for a given model, visualized via quadrant scatter plots to identify models that are both high-performing and stable.

Key Contributions

• Comprehensive Coverage: Nsanku is the first large-scale evaluation covering 43 Ghanaian languages, 42 of which have never appeared in a major multilingual MT benchmark.
• Infrastructure: The project releases a community-extensible evaluation infrastructure, including the full pipeline code, data collection scripts, and a Google Colab notebook, allowing researchers to contribute new models or language subsets.
• Multi-Dimensional Analysis: Beyond aggregate accuracy, the study introduces a cross-lingual consistency dimension to determine if model performance is stable across languages or concentrated on a small subset.
• Zero-Shot Benchmarking: It provides the first rigorous assessment of zero-shot translation capabilities for this specific linguistic region, addressing the gap between proprietary and open-weight model performance.

Results

The evaluation, completed on January 29, 2026, yielded the following findings:

• Model Performance: Proprietary frontier models significantly outperformed open-weight alternatives.
  • Top Performers: gemini-2.5-flash achieved the highest average score (26.88), followed by claude-sonnet-4-5 (24.87) and gpt-4.1 (23.20).
  • Open-Weight Leaders: kimi-k2-instruct-0905 led the open-weight category with an average score of 20.87, followed by deepseek-v3.1 (19.29).
  • Performance Gap: There is a significant gap (approx. 6 points) between the best proprietary model and the best open-weight model, though the gap is narrowing compared to historical trends.
• Metric Discrepancy: Across all models, chrF scores substantially exceeded BLEU scores. For the lowest-performing model (mistral-7b-instruct-v0.3), the gap was 15.76 points (BLEU 4.15 vs. chrF 19.91). This suggests that models preserve character-level morphology better than exact word-level matches, validating chrF as a more appropriate metric for morphologically rich Ghanaian languages.
• Language Variance: Performance varied widely across languages. Siwu achieved the highest per-language average (25.73), while Nkonya scored the lowest (11.65). Interestingly, some languages with smaller speaker populations (e.g., Siwu, Chumburung) outperformed widely spoken languages like Twi (21.82), suggesting that the quality of the reference Bible translation in the scriptural domain is a critical factor.
• Consistency Analysis (Critical Finding): No model and no language occupied the "Leaders" quadrant (high performance and high consistency).
  • Top models fell into the "High Performance but Inconsistent" quadrant, meaning their quality on any specific language is unpredictable.
  • No Ghanaian language currently receives high-quality and consistent translation across the range of evaluated models.

Significance and Claims

The paper claims that Nsanku establishes a rigorous baseline for what current LLMs can and cannot do for Ghanaian languages. Its primary significance lies in exposing the reliability gap: despite impressive aggregate scores from frontier models, the absence of any model in the "Leaders" quadrant indicates that current LLMs are not yet reliably usable for Ghanaian language translation at scale.

The authors emphasize that the low scores reflect resource scarcity and training data imbalances rather than intrinsic linguistic complexity. They argue that without systematic benchmarks like Nsanku, developers cannot make evidence-based deployment decisions, and policymakers cannot identify critical capability gaps. The study concludes that while open-weight development is beginning to close the gap with proprietary models, the field requires further investment in data and evaluation to achieve reliable, consistent translation for African languages. The authors explicitly caution that these results are specific to the scriptural domain and may not generalize to conversational or journalistic text without further evaluation.