Left Behind: Cross-Lingual Transfer as a Bridge for Low-Resource Languages in Large Language Models

1. Problem Statement

Despite the rapid deployment of Large Language Models (LLMs) as general-purpose tools, their capabilities are profoundly unequal across languages. While high-resource languages (like English) benefit from vast training corpora and rigorous benchmarks, Low-Resource Languages (LRLs)—specifically those with agglutinative morphology and non-Latin scripts like Kazakh and Mongolian—suffer from degraded accuracy, unreliable fluency, and a lack of culturally grounded knowledge.

Existing evaluations often rely on machine-translated benchmarks that fail to capture cultural nuances or focus on well-studied language families. Furthermore, while Cross-Lingual Transfer (CLT) (prompting models to translate to English, reason, and translate back) has been proposed as a workaround, its effectiveness across different model architectures (English-first vs. Bilingual vs. Multilingual) remains untested.

2. Methodology

Benchmark Construction

The authors constructed a controlled benchmark consisting of 2,000 responses generated by 8 LLMs across 50 hand-crafted questions.

• Languages: English (High-Resource), Kazakh (19M speakers), and Mongolian (6M speakers).
• Question Categories: Five categories with 10 questions each:
  1. Neutral Factual
  2. Neutral Reasoning
  3. Neutral Technical
  4. Kazakh-aligned Cultural
  5. Mongolian-aligned Cultural
• Design: Each question has human-authored parallel versions in all three languages. Reference answer sheets allow for multiple acceptable responses to handle ambiguity.

Experimental Conditions

Five distinct conditions were tested for every model-question pair:

1. C1 (English Baseline): Questions and responses in English.
2. C2 (Kazakh Direct): Questions in Kazakh, response in Kazakh.
3. C3 (Kazakh CLT): Three-step pipeline in Kazakh: (1) Translate to English, (2) Reason/Answer in English, (3) Translate back to Kazakh.
4. C4 (Mongolian Direct): Questions in Mongolian, response in Mongolian.
5. C5 (Mongolian CLT): Same three-step pipeline as C3 but for Mongolian.

Models Evaluated

The 8 models were categorized by their training data profiles:

• English-First (5 models): Claude Opus 4.5, GPT-5.2, Gemini 3 Pro, Grok 4.1, Llama 4 Maverick. (Primarily English training; multilingualism is secondary).
• Bilingual (2 models): Qwen3, DeepSeek V3.2. (Substantial Chinese + Multilingual data alongside English; hypothesized to handle non-Latin scripts better).
• Multilingual (1 model): Aya Expanse. (Explicitly trained for 100+ languages, including Turkic families).

Evaluation Metrics

Responses were scored semi-automatically (using an LLM-as-judge paradigm) on a 0–6 scale based on three dimensions:

1. Accuracy (0–2): Factual correctness and logical soundness.
2. Fluency (0–2): Grammatical correctness and naturalness. Strict penalty: If the model outputs the wrong language (e.g., Kyrgyz instead of Kazakh), Fluency = 0.
3. Completeness (0–2): Whether the answer addresses all parts of the prompt.

3. Key Results

Performance Gap

There is a consistent and significant performance gap between English and LRLs:

• English Baseline (C1): 90.7% average score.
• Direct LRL (C2/C4): Scores dropped to 76.9% (Kazakh) and 74.0% (Mongolian).
• The Gap: A 13.8 to 16.7 percentage point deficit for LRLs compared to English.

The "Fluency Illusion"

Models maintained high surface-level fluency while producing factually incorrect content.

• Kazakh: Accuracy dropped 13.0 points, but Fluency dropped only 11.0 points.
• Mongolian: Accuracy dropped 16.0 points, while Fluency remained high at 87.5%.
• Implication: LRL users receive text that looks correct but contains hallucinations, posing a deceptive risk.

Impact of Cross-Lingual Transfer (CLT)

The effectiveness of CLT was architecture-dependent:

• English-First Models: Showed negligible or negative gains from CLT (e.g., -1.1pp for Kazakh). These models likely already reason internally in English; forcing an explicit translation pipeline adds latency without improving accuracy.
• Bilingual Models: Showed selective gains (+2.2pp for Kazakh, +4.3pp for Mongolian). The structured "translate-reason-translate" pipeline compensated for weaker reasoning capabilities in the target language.
• Multilingual Model (Aya Expanse): Performed disastrously in LRLs.
  • English: 82.3%
  • Kazakh Direct: 15.7%
  • Mongolian Direct: 4.7%
  • Critical Failure: The model frequently generated Kyrgyz instead of Kazakh and garbled Mongolian, despite being explicitly trained on Turkic languages.

Latency and Efficiency

• Response Time: Direct LRL prompting increased latency significantly (e.g., Mongolian Direct: 25.0s vs. English: 7.3s).
• CLT Overhead: The CLT pipeline further increased latency (up to 35.4s for Kazakh CLT) and reduced response length, indicating a trade-off between potential accuracy gains and efficiency.

4. Key Contributions

1. Controlled Benchmark: A novel, human-crafted benchmark specifically for Turkic and Mongolic languages, avoiding the pitfalls of machine-translated datasets.
2. Architecture-Specific Analysis: The first study to systematically compare how English-first, Bilingual, and Multilingual architectures handle LRLs and CLT strategies.
3. Identification of the "Fluency Illusion": Empirical evidence that LRL outputs often appear fluent but are factually unreliable, a critical safety concern for end-users.
4. CLT Viability: Demonstrated that CLT is not a universal fix; it acts as "scaffolding" only for models with partial multilingual competence (Bilingual), failing to help English-first models and exacerbating failures in poorly trained Multilingual models.

5. Significance and Conclusion

The paper concludes that current LLMs systematically underserve low-resource language communities. The "multilingual" label on some models does not guarantee reliable performance on specific low-resource languages, particularly within closely related language families (e.g., confusing Kazakh with Kyrgyz).

Implications:

• Mitigation is not universal: Strategies like CLT must be tailored to the model's architecture.
• Evaluation needs: Aggregate multilingual benchmarks are insufficient; language-specific evaluations are urgent.
• Future Work: The authors suggest that targeted fine-tuning on small LRL corpora may be more effective than prompting strategies alone to close the performance gap.

This study serves as a critical warning to developers and policymakers: deploying LLMs in low-resource contexts without architecture-specific validation risks providing users with fluent but fundamentally incorrect information.