KyrgyzLLM-Bench: Benchmarking Kyrgyz Language Understanding

Technical Summary: KyrgyzLLM-Bench

Problem Statement
Evaluating Large Language Models (LLMs) across diverse languages remains a significant challenge, particularly for less-resourced languages like Kyrgyz. Current multilingual benchmarks predominantly rely on machine-translated English datasets, which often introduce "translationese," subtle errors, and cultural misalignments that obscure linguistic specificity and compromise evaluation reliability. For Kyrgyz, a Turkic language with agglutinative morphology and limited representation in pretraining corpora, there is a critical scarcity of high-quality, natively authored evaluation data. This gap hinders systematic research and the development of reliable language technologies for the region.

Methodology
The authors introduce KyrgyzLLM-Bench, a systematic, large-scale benchmark suite designed to evaluate LLMs in Kyrgyz through a hybrid approach combining native authorship and carefully curated translations. The benchmark comprises three core components:

1. KyrgyzMMLU: A large-scale, natively authored multiple-choice question-answering dataset containing 7,977 items. Derived from the Kyrgyz General Republican Testing (GRT), it covers eight school subjects (Mathematics, Biology, Physics, Chemistry, Kyrgyz Literature, Geography, Kyrgyz History, Kyrgyz Language) and a Medicine specialization. Items were written by curriculum experts and aligned with the national education registry.
2. KyrgyzRC: A native reading comprehension dataset consisting of 400 multiple-choice questions based on authentic Kyrgyz texts from four domains: Kyrgyz Wikipedia, national news, literature, and school-level mathematics. The dataset was constructed by native Kyrgyz speakers (students and supervisors) through a multi-stage pipeline involving item creation, domain-level curation, and final linguistic review.
3. Translated Benchmarks: Manually post-edited Kyrgyz versions of four established English benchmarks: WinoGrande (pronoun resolution), HellaSwag (event continuation), BoolQ (reading comprehension), and TruthfulQA (factuality). These were translated using an ensemble of LLMs followed by rigorous manual review by native linguists to ensure cultural appropriateness and linguistic fidelity.

The study evaluated 26 open-source and closed-source LLMs (including Qwen, Gemma, Llama, GPT, Claude, and others) under both zero-shot and few-shot (5-shot or 10-shot) settings. The evaluation utilized the Lighteval framework, measuring accuracy via text-based answer parsing. The authors also established parallel English baselines to analyze cross-lingual transfer.

Key Results

• Model Performance: Performance generally scales with model capacity. Among open-source models, Qwen3-8B achieved the highest average accuracy in few-shot Kyrgyz evaluation (52.7%), followed by Llama-3.1-8B-Instruct (50.3%). Proprietary models showed higher performance, with Claude 4.5 Sonnet leading in both zero-shot (74.82%) and few-shot (77.06%) settings.
• Cross-Lingual Transfer: Model rankings observed in English largely transferred to Kyrgyz for WinoGrande and BoolQ, and to a lesser extent for MMLU. However, HellaSwag exhibited a substantial performance gap between English and Kyrgyz, with Kyrgyz scores significantly lower.
• Few-Shot Effects: The impact of few-shot prompting was inconsistent. For open-source models, few-shot prompting improved performance on KyrgyzRC and BoolQ but showed limited gains or even degradation on HellaSwag. For proprietary models, few-shot prompting often yielded stagnant or negative gains, particularly on translated tasks like BoolQ, likely due to performance saturation in zero-shot settings.
• Subject Variance: Models performed better on cross-lingually transferable subjects (Mathematics, Physics) compared to culture-specific subjects (Kyrgyz Literature, Kyrgyz Language), highlighting the limitations of pretraining data regarding specific cultural knowledge.

Key Contributions

1. First Systematic Evaluation: This work presents the first large-scale, systematic evaluation of LLMs in Kyrgyz, addressing the lack of native evaluation resources.
2. Native Dataset Construction: The authors introduce KyrgyzMMLU and KyrgyzRC, the first natively authored, expert-validated benchmarks for Kyrgyz, moving beyond the limitations of purely translated datasets.
3. Translation Artifact Analysis: The study provides empirical evidence supporting the "plausibility-shift hypothesis," demonstrating that translated event-continuation tasks (like HellaSwag) suffer from unpredictable changes in naturalness and coherence, leading to unreliable cross-lingual measurements.
4. Open Release: All datasets, evaluation code, and per-model results are publicly released and integrated into the Lighteval framework to support future Kyrgyz NLP research.

Significance and Claims
The paper claims that KyrgyzLLM-Bench fills a critical gap in the evaluation of less-resourced, morphologically rich languages. It demonstrates that while modern instruction-tuned models show generalization capabilities, their performance on natively authored Kyrgyz tasks remains substantially below English baselines. The authors emphasize that evaluation methodology significantly influences outcomes; specifically, relying on translated prompts for culturally sensitive tasks can lead to misleading conclusions about model reasoning capabilities.

The study concludes that few-shot prompting in low-resource, translated-prompt scenarios should not be assumed to be uniformly beneficial and must be interpreted with caution. The work advocates for the development of culturally grounded, natively authored benchmarks to support equitable progress in LLM development for Central Asian languages and to improve the accessibility of AI technologies across diverse linguistic communities.