💬 NLP

ChiKhaPo: A Large-Scale Multilingual Benchmark for Evaluating Lexical Comprehension and Generation in Large Language Models

The paper introduces ChiKhaPo, a large-scale multilingual benchmark covering over 2700 languages with eight subtasks designed to evaluate the lexical comprehension and generation abilities of large language models, revealing that even state-of-the-art models struggle with basic linguistic competence in the vast majority of the world's written languages.

Original authors: Emily Chang, Niyati Bafna

Published 2026-06-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Emily Chang, Niyati Bafna

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, super-smart library of books written in thousands of different languages. You hire a brilliant new librarian (a Large Language Model, or LLM) to help you find specific words and translate them. You want to know: Does this librarian actually understand the words, or are they just guessing based on the few languages they've studied the most?

This paper introduces a new test called ChiKhaPo (pronounced Chi-Kha-Po) to answer exactly that question. The name comes from a saying meaning "step by step," because the authors believe we need to take small, careful steps to understand how these AI models really work across the world's languages.

Here is the breakdown of what they did, using some everyday analogies:

1. The Problem: The "VIP Lounge" of Languages

Currently, most tests for AI are like VIP lounges. They only let in a few dozen "High-Resource Languages" (like English, Spanish, or French). These are languages with tons of data on the internet.

The Reality: There are over 3,800 written languages in the world. The vast majority are locked out of these VIP lounges.
The Gap: We don't know if the AI can handle the other 3,700+ languages. It might be a genius in English but completely lost when asked to translate a word in a low-resource language.

2. The Solution: The "Massive Multilingual Exam" (ChiKhaPo)

The authors built a massive exam that covers 2,700+ languages. Instead of asking the AI to write a complex essay or solve a math problem (which are hard tasks), they focused on the basics: Lexical Competence.

The Analogy: Think of it like testing a student's vocabulary before asking them to write a novel. Can they recognize a word? Can they say what it means? Can they use it in a sentence?

The exam has 8 different sections (subtasks) to test these skills from different angles:

Word Translation: "What is the word for 'rain' in Malay?" (Direct translation).
Word Translation with Context: "In this story about a storm, what does the word 'ujan' mean?" (Using context clues).
Translation-Conditioned Modeling: The AI is given a sentence in one language and has to predict the next word in the translation. (Like a "fill-in-the-blank" game).
Bag-of-Words Translation: The AI translates a whole sentence, and the test checks if it got the individual words right, even if the sentence structure is a bit messy.

3. The Results: The AI Struggles with the Basics

The authors tested 6 of the smartest AI models available today on this exam.

The Finding: Even the best models struggled significantly, especially with low-resource languages.
The "Comprehension vs. Generation" Gap: The models were better at understanding a word (reading it and knowing what it means) than generating it (saying or writing the word themselves).
- Analogy: It's like a person who can read a menu in a foreign language and know what "soup" is, but when asked to order it, they freeze and can't say the word.
The "Rich vs. Poor" Gap: The models performed much better on languages that have lots of data (High-Resource) compared to those with very little data (Low-Resource). The performance gap was huge.

4. Why This Matters (According to the Paper)

The paper argues that we can't just assume AI is "multilingual" because it works well in English.

The "Proxy" Discovery: They found that if an AI is good at translating single words (the simple test), it's usually good at translating full sentences (the complex test). This means the simple word test is a cheap and easy way to check if an AI is ready for a harder job.
The Goal: The authors want to shine a light on language inequality. Right now, NLP (Natural Language Processing) is unfair because it ignores thousands of languages. ChiKhaPo is a tool to force researchers to pay attention to these neglected languages and build better, fairer AI.

Summary

ChiKhaPo is a giant, step-by-step vocabulary test for AI across 2,700+ languages. It reveals that even the smartest AI models are currently "word-blind" for most of the world's languages. They can often understand a word but struggle to produce it, and they perform terribly on languages that don't have a lot of data online. The authors hope this test will encourage the AI community to stop focusing only on the "VIP" languages and start building models that truly understand the whole world.

Technical Summary: ChiKhaPo

Problem Statement

Current benchmarks for Large Language Models (LLMs) are predominantly restricted to high-resource languages (HRLs) and focus on higher-order tasks such as complex reasoning and content generation. Consequently, there is a significant gap in evaluating the basic linguistic competence of LLMs across the world's 3,800+ written languages. Existing multilingual benchmarks, such as FLORES+, cover only a fraction of these languages (e.g., 212 languages), leaving the vast majority of the world's languages unassessed even for fundamental capabilities like word-level understanding and generation. This lack of evaluation hinders the identification of language inequities in Natural Language Processing (NLP).

Methodology

The authors introduce ChiKhaPo, a massively multilingual benchmark designed to evaluate lexical comprehension and generation abilities in LLMs. The benchmark is constructed using existing public resources, including lexicons, monolingual data, and bitext, allowing for coverage of over 2,700 languages.

Benchmark Structure

ChiKhaPo consists of eight subtasks derived from four core tasks, each evaluated in two directions:

Comprehension ( $X \to \text{model}$ ): Evaluating the model's ability to understand a word in a target language $X$ (often by translating it to English).
Generation ( $\text{model} \to X$ ): Evaluating the model's ability to generate a word in target language $X$ given an English prompt.

The four core tasks are:

Word Translation (WT): Direct prompting to translate a single word. This task relies solely on lexicons and covers 2,746 languages.
Word Translation with Context (WTWC): Translating a word given a source sentence context. This requires monolingual data and covers 525 languages.
Translation-Conditioned Language Modeling (TCLM): Measuring the generation probability of a target word given a source sentence and partial target context. This uses parallel sentence pairs (bitext) and covers 211 languages.
Bag-of-Words Machine Translation (BOW MT): Evaluating whether specific target words appear in a sentence-level translation output, regardless of syntactic order. This also uses bitext and covers 211 languages.

Scoring and Evaluation

The benchmark employs word-level scoring metrics. For binary tasks (WT, WTWC), correctness is determined via exact matches, inflection handling, substring matching, and synonymy (using WordNet for English outputs). For probabilistic tasks (TCLM), the metric is the generation probability of the correct word. For BOW MT, scores are aggregated based on the presence of target words in the output.

The authors evaluated six state-of-the-art (SOTA) multilingual LLMs: aya-101, aya-23-8b, bloomz-7b1-mt, falcon-7b-instruct, gemma-2b-it, and Llama-3.1-8B-Instruct. Due to computational constraints, a "lite" version of the benchmark was used, sampling up to 300 vocabulary entries per language for WT/WTWC and 30% of available data for sentence-level tasks.

Key Results

Performance Gaps: All six evaluated models demonstrated significant room for improvement, particularly on low-resource languages (LRLs). The benchmark successfully acts as a challenging measure of multilingual performance.
Directional Disparity: Models consistently performed better in the comprehension direction ( $X \to \text{model}$ ) than in the generation direction ( $\text{model} \to X$ ), highlighting a gap between Natural Language Understanding (NLU) and Natural Language Generation (NLG) in multilingual settings.
Resource and Family Correlation: Performance was strongly correlated with language resource levels (proxied by Wikipedia document counts) and language families. Indo-European languages consistently outperformed underrepresented families like Austronesian and Atlantic-Congo. The relationship between resource level and performance was found to be roughly logarithmic.
Model Differences: The aya-101 model achieved the highest average scores on five of the eight subtasks. The authors attribute this to its encoder-decoder architecture, larger parameter count (13B), and instruction tuning on 101 languages.
Correlation with MT: Word Translation (WT) scores showed a strong linear correlation with sentence-level machine translation (BLEU) scores on FLORES+ data ( $r \approx 0.87$ for $X \to \text{model}$ ). This suggests WT can serve as a cost-effective proxy for evaluating machine translation capabilities in the absence of extensive bitext.

Significance and Claims

The paper positions ChiKhaPo as a critical step toward addressing language inequity in NLP. Its primary contributions are:

Massive Scale: It provides the first lexical competence benchmark covering 2,700+ languages, surpassing the language coverage of any existing benchmark by an order of magnitude.
Granular Evaluation: It shifts focus from complex reasoning to "atomic" word-level competence, allowing researchers to track the fundamental linguistic capabilities of LLMs in specific languages.
Resource Efficiency: By leveraging existing lexicons and monolingual data, it enables the evaluation of languages for which no parallel corpora exist.

The authors modestly claim that while the benchmark relies on public resources (which can be noisy, particularly PanLex), it successfully highlights the pressing need for improved multilingual evaluation. They hope this work will encourage the collection of better lexical resources for low-resource languages and promote the massively multilingual benchmarking of LLMs as a necessary step toward more equitable AI systems.