Evaluating Modern Large Language Models on Low-Resource and Morphologically Rich Languages:A Cross-Lingual Benchmark Across Cantonese, Japanese, and Turkish

1. Problem Statement

While Large Language Models (LLMs) have achieved near-human performance in high-resource languages like English, their effectiveness in low-resource and morphologically rich languages remains under-explored. Existing benchmarks often suffer from:

• Underrepresentation: Many languages with complex morphology (e.g., Turkish) or unique script/dialectal features (e.g., Cantonese, Japanese) are missing or poorly covered.
• Cultural Blindness: Models often fail to grasp cultural nuances, honorifics, and idioms specific to non-Western contexts.
• Evaluation Gaps: Automated metrics (BLEU, ROUGE) often fail to capture fluency, factual accuracy, and cultural appropriateness in low-resource settings.

The authors aim to answer: How well do state-of-the-art LLMs generalize to morphologically complex and culturally specific languages, and do open-source models close the gap with proprietary giants in these domains?

2. Methodology

Benchmark Design

The authors constructed a new cross-lingual benchmark covering three typologically distinct languages:

1. Turkish: Highly agglutinative with rich morphology (root + affix combinations), testing compositional generalization.
2. Japanese: Complex writing system (Kanji, Hiragana, Katakana) and context-dependent grammar (honorifics, topic drop).
3. Cantonese: A low-resource language (especially in written form) with distinct colloquial grammar and unique vocabulary/idioms.

Four Evaluation Tasks:

1. Open-Domain QA: 100 questions per language testing knowledge retrieval and factuality.
2. Document Summarization: 50 document-summary pairs per language (sourced from Wikinews, MLSUM, and HK news) testing coherence and information condensation.
3. English-to-X Translation: 100 sentences per language (news, dialogue, encyclopedia) testing translation quality.
4. Culturally Grounded Dialogue: 15 multi-turn scenarios per language where the model must respond to culturally specific inputs (e.g., using honorifics in Japanese, understanding idioms in Cantonese).

Models Evaluated

Seven LLMs were evaluated, spanning proprietary and open-source, and varying scales:

• Proprietary: GPT-4o, GPT-4, Claude 3.5 Sonnet.
• Open-Source (Large): LLaMA 3.1 (70B), Mistral Large 2 (~128B).
• Open-Source (Small): LLaMA-2 Chat 13B, Mistral 7B Instruct.

Evaluation Protocol

• Automated Metrics: Exact-match accuracy (QA), ROUGE/BERTScore (Summarization), BLEU/COMET (Translation).
• Human Evaluation: Three native speakers per language rated outputs on a 5-point Likert scale for Fluency, Factual Accuracy, and Cultural Appropriateness. This was crucial for dialogue and translation where automated metrics are brittle.

3. Key Contributions

1. New Benchmark: Introduction of a comprehensive, cross-lingual benchmark specifically targeting low-resource and morphologically rich languages (Turkish, Japanese, Cantonese) across four diverse tasks.
2. Comprehensive Model Comparison: The first study to compare the latest generation of models (GPT-4o, Claude 3.5, LLaMA 3.1) on these specific languages.
3. Rigorous Evaluation Framework: A hybrid approach combining automated metrics with extensive human evaluation to capture cultural and linguistic nuances often missed by standard scores.
4. Open Release: All prompts, evaluation data, and human annotations are released to foster reproducibility.

4. Key Results

Performance Hierarchy

• Top Tier: GPT-4o emerged as the overall leader, showing robust multilingual performance, particularly in Cantonese and culturally nuanced dialogue. GPT-4 and Claude 3.5 Sonnet followed closely.
• Mid Tier: LLaMA 3.1 (70B) and Mistral Large 2 performed competitively on Turkish and Japanese (approaching proprietary models) but lagged significantly on Cantonese.
• Bottom Tier: Smaller models (LLaMA-2 13B, Mistral 7B) struggled substantially, showing poor fluency, high hallucination rates, and a failure to handle morphological complexity or cultural context.

Task-Specific Findings

• Morphology (Turkish): Even top models occasionally failed to generate correct case suffixes or plural forms, indicating that LLMs are not yet fully systematic in morphological generalization.
• Cultural Nuance (Japanese/Cantonese):
  • Top models (GPT-4o) successfully adapted tone (honorifics) and understood idioms.
  • Smaller models often defaulted to generic, culture-agnostic responses or used incorrect registers (e.g., informal speech in formal contexts).
  • Cantonese Specifics: GPT-4o avoided the common error of switching to Mandarin or English, whereas other models frequently failed to use Cantonese-specific particles (e.g., using de instead of ge).
• Translation: GPT-4o achieved the highest BLEU and COMET scores. Smaller models often produced ungrammatical or literal translations.

Quantitative vs. Qualitative

• Automated metrics (BLEU/ROUGE) correlated moderately with human judgments but often penalized fluent paraphrasing or failed to detect cultural insensitivity.
• Human evaluation was indispensable, revealing that models with similar automated scores could differ vastly in cultural appropriateness.

5. Significance and Implications

• Resource Disparity: The study highlights a persistent gap between proprietary and open-source models, especially in low-resource domains. While open models are closing the gap in high-resource languages, they still trail significantly in dialects like Cantonese.
• Cultural Intelligence: Current models are beginning to grasp cultural context, but this capability appears to emerge only at larger scales and with specific training. Smaller models lack the "cultural common sense" required for polite or idiomatic interaction.
• Future Directions:
  • Data: There is an urgent need for more diverse, high-quality training data for low-resource languages and dialects.
  • Architecture: Future models may need morphology-sensitive architectures or specialized tokenization strategies.
  • Evaluation: The field must move beyond automated metrics to include rigorous human evaluation for cultural alignment.

In conclusion, while modern LLMs have made strides in multilingual capabilities, significant challenges remain in handling morphological complexity and cultural nuance, particularly for low-resource languages. The authors' benchmark provides a critical tool for measuring progress and guiding the development of more equitable and linguistically generalizable AI systems.