AmharicStoryQA: A Multicultural Story Question Answering Benchmark in Amharic

Technical Summary: AmharicStoryQA

Problem Statement
Current evaluations of Large Language Models (LLMs) often treat language and culture as synonymous, using performance metrics as a proxy for understanding a specific language. This approach overlooks meaningful cultural variations that exist within a single language community. Specifically, existing benchmarks for low-resource languages like Amharic frequently fail to capture regional diversity, dialectal nuances, and domain-specific content arising from different governmental structures, histories, and social norms. Furthermore, there is a scarcity of generation-focused, long-sequence datasets designed to simulate real-life model usage in these contexts, leaving a significant blind spot in understanding how LLMs handle extensive narratives in Ethiopian languages.

Methodology
To address these gaps, the authors introduce AmharicStoryQA, a multicultural story-based question answering benchmark. The methodology involves the following key steps:

• Data Collection: The dataset is constructed from 244 pre-translated stories and folktales collected from nine distinct regions of Ethiopia (including Afar, Amhara, Benishangul-Gumuz, Harar, Oromia, Somalia, SNNP zones, and Tigray).
• Question Generation: Using GPT-4.1, the authors generated five distinct multiple-choice questions (MCQA) and one correct answer with three distractors (biased by world knowledge, unrelated, or factually false) for each story.
• Quality Control:
  • Manual Rating: Three human annotators evaluated questions for correctness, faithfulness, linguistic quality, and comprehension depth using Gwet's AC1 for inter-rater reliability.
  • Translation: Experienced annotators translated questions and answers into Amharic, preserving narrative style and cultural relevance.
  • Automated Scoring: The SSA-COMET metric was used to flag low-quality machine translations (score < 0.65) for manual inspection, ensuring high fidelity to the original stories.
• Dataset Split: The final benchmark consists of 571 training and 649 test examples, split based on SSA-COMET scores to prevent translation inconsistencies from disproportionately influencing evaluation.
• Evaluation Protocol: The authors evaluated seven open-source LLMs (including Gemma, Llama, Aya, and Command R) with context lengths up to 128k tokens. Evaluations covered both MCQA (using log-likelihood) and open-ended generation (using chrF). Additionally, supervised fine-tuning (SFT) was performed on Llama 3.1 8B Instruct using LoRA on English-only, Amharic-only, and combined datasets.

Key Contributions

1. AmharicStoryQA Benchmark: A multicultural, long-sequence QA dataset in Amharic and English, comprising 1,220 examples derived from 244 stories across nine Ethiopian regions. It includes both MCQA and open-ended generation tasks.
2. Empirical Analysis of Regional Variation: The study demonstrates that despite shared linguistic characteristics, region-specific content substantially influences evaluation outcomes. It reveals that existing LLMs exhibit significant narrative understanding gaps and pronounced regional performance differences.
3. Fine-Tuning Insights: The paper investigates the impact of culturally grounded training data, showing that supervised fine-tuning yields uneven improvements across regions and evaluation settings, highlighting the necessity of targeted data for low-resource languages.

Results and Analysis

• Narrative Comprehension Gap: In zero-shot evaluations, most models (except Gemma) exhibited a sharp performance collapse on Amharic MCQA tasks (dropping to 0.20–0.35 accuracy) compared to their high performance on English tasks. This suggests a structural limitation in Amharic narrative understanding rather than a failure of instruction following.
• Generative vs. Comprehension: While generative evaluation (chrF) showed a more gradual degradation than MCQA, models still underperformed significantly on Amharic narratives. Notably, providing English prompts for Amharic narratives improved surface-level fluency but did not correlate with higher comprehension accuracy.
• Regional Differences: Semantic density analysis (via PCA) showed that while Amharic stories share a common semantic space, regional contours reveal distinct cultural motifs. Performance varied across regions, with some models showing up to ±10 chrF point differences or 20% accuracy variance depending on the region.
• Impact of Fine-Tuning: SFT on the AmharicStoryQA dataset significantly improved MCQA accuracy for Llama 3.1 8B Instruct (nearly 40% improvement in Amharic tasks) and reduced the performance gap between English and Amharic prompting. Fine-tuning also stabilized results across regions for Amharic narratives, though regional differences remained more pronounced for English narratives even after training.

Significance
The paper argues that standard language-level evaluations are insufficient for assessing LLMs in low-resource, multicultural contexts. By introducing a benchmark grounded in regional diversity, the authors demonstrate that cultural variation within a single language meaningfully affects story comprehension. The findings emphasize the need for culturally grounded benchmarks and training data to accurately assess and improve narrative understanding in languages like Amharic, moving beyond the assumption that linguistic proficiency equates to cultural understanding. The work serves as a call to action for the community to develop evaluation frameworks that account for intra-language diversity rather than treating languages as monolithic cultural entities.