Cultural Benchmarking of LLMs in Standard and Dialectal Arabic Dialogues

1. Problem Statement

Current Large Language Model (LLM) evaluation for Arabic culture suffers from two critical limitations:

• MSA-Centric Bias: Existing benchmarks (e.g., ArabCulture) rely almost exclusively on Modern Standard Arabic (MSA) and isolated, single-turn multiple-choice questions. This overlooks the fact that daily communication in the Arab world occurs primarily in diverse regional dialects, which encode specific cultural norms, social identities, and pragmatic nuances distinct from MSA.
• Lack of Conversational Context: Cultural reasoning often unfolds across multi-turn dialogues where speakers must interpret implicit norms and maintain pragmatic coherence. Single-turn evaluations fail to capture this dynamic, potentially overestimating model capabilities by testing them in a register (MSA) that is less culturally laden than authentic usage.

The core research question is: Can models that perform adequately on MSA-based cultural questions actually apply this knowledge in natural, dialect-mediated, multi-turn dialogues?

2. Methodology

A. Dataset Construction: ArabCulture-Dialogue

The authors introduce ArabCulture-Dialogue, the first parallel conversational dataset covering 13 Arabic-speaking countries (including UAE, Egypt, Morocco, etc.) in both MSA and their respective local dialects.

• Scale: 3,471 unique dialogue instances (6,942 total when counting both MSA and Dialect versions), spanning 12 daily-life topics (e.g., weddings, food, parenting) and 54 fine-grained subtopics.
• Construction Pipeline:
  1. Generation: Initial drafts generated by GPT-4o based on the existing ArabCulture premises.
  2. Human Curation: 26 native speakers (fluent in MSA and local dialects, with >10 years of residency) manually revised the MSA dialogues for naturalness and cultural accuracy.
  3. Dialect Localization: Annotators translated MSA dialogues into specific country dialects, avoiding literal translation to ensure authentic conversational flow.
  4. Quality Control: A multi-stage process involving cross-review, style consistency checks (ensuring answer options have similar length/tone to prevent superficial pattern matching), and a final independent QC.
• Statistics: The dataset contains ~343,804 words. Dialectal data exhibits a larger vocabulary size than MSA due to the lack of standardized orthography in dialects.

B. Evaluation Tasks

The dataset is used to benchmark LLMs across three distinct tasks:

1. Dialogue-Based Cultural Reasoning (MCQ): Models must select the culturally appropriate response from three options in a multi-turn dialogue. This tests cultural commonsense reasoning in context.
2. Dialect Translation: Models translate multi-turn dialogues between MSA and specific country dialects (bidirectional). Metrics include BLEU, BERTScore, ALDi (Arabic Level of Dialectness), and an LLM-as-a-Judge framework.
3. Dialect Steering: Models are instructed to continue a dialogue in a specific target dialect (or MSA). Performance is measured by generation quality (LLM-as-a-Judge) and dialect identity accuracy (using GlotLID for language identification).

C. Models Evaluated

The study evaluated a wide range of models:

• Arabic-Centric Open-Weight: Jais, ALLaM, SILMA, Cohere-Arabic, Fanar, Hala (7B–9B parameters).
• Multilingual Open-Weight: Gemma, Qwen, LLaMA (8B–9B parameters).
• Proprietary Models: GPT-5 and Gemini-2.5-Pro.

3. Key Contributions

1. First Parallel MSA-Dialect Dialogue Dataset: ArabCulture-Dialogue is the first resource to benchmark Arabic cultural reasoning across 13 countries in both MSA and dialects within a conversational setting.
2. Comprehensive Benchmarking Framework: Defined three complementary tasks (MCQ, Translation, Steering) to evaluate cultural reasoning, cross-register linguistic competence, and controlled generation.
3. Empirical Evidence of Dialectal Gaps: Provided extensive experimental results demonstrating that current open-weight models struggle significantly with dialectal cultural reasoning compared to MSA, often failing to transfer cultural knowledge from standard to dialectal settings.

4. Key Results and Analysis

A. Performance Degradation in Dialects

• MSA vs. Dialect Gap: All models performed significantly worse on dialectal dialogues compared to MSA across all three tasks.
• Proprietary vs. Open-Weight: Proprietary models (GPT-5, Gemini-2.5-Pro) achieved high accuracy (~94-95%) in MCQ tasks for both MSA and dialects. In contrast, open-weight models (even 9B parameter ones) showed substantial drops, with some performing near random guessing (33%) on dialectal MCQs.
• Country Specificity: Models performed better on non-country-specific dialogues than on country-specific ones. North African dialects presented the highest difficulty, while Jordanian and Palestinian dialects were slightly easier for models to infer.

B. Translation Challenges

• Directionality: Models translated Dialect-to-MSA better than MSA-to-Dialect.
• Dialect Correctness: While semantic adequacy (meaning) was often preserved, Register (dialectal correctness) scores were low for open models. Many models generated fluent text that was semantically correct but linguistically "pan-Arabic" or MSA-like, failing to capture specific country-level dialect features.
• Fine-Tuning Impact: Supervised Fine-Tuning (SFT) on multilingual models improved fluency and terminology but did not fully close the gap with zero-shot Arabic-centric models. SFT helped some models generate more distinctive dialectal cues (e.g., in Moroccan Darija) but struggled with others.

C. Dialect Steering Limitations

• Identity Failure: Even when models generated fluent responses, they often failed to match the strict ISO code of the target dialect. For example, models generating UAE dialect were often identified by GlotLID as "Gulf" (afb) or "Najdi" (ars) rather than the specific UAE target, highlighting the difficulty of fine-grained dialect control.
• Trade-offs: SFT improved controllability but sometimes reduced the model's ability to stay in MSA when requested, indicating a trade-off between output quality and strict dialect adherence.

5. Significance and Future Directions

• Cultural Competence is Contextual: The study proves that cultural reasoning is not a static knowledge base but a dynamic skill that degrades when the linguistic register shifts from MSA to dialect.
• Limitations of Current Open Models: Current open-weight LLMs (7B–9B) are insufficient for culturally grounded, dialect-rich applications. They lack the nuanced understanding of local norms required for real-world deployment in the Arab world.
• Future Work: The authors call for:
  • Dialect-aware pretraining and instruction tuning.
  • Expansion of datasets to cover more Arab countries and intra-country variations.
  • Development of models that better integrate cultural knowledge with conversational dynamics.

Conclusion: ArabCulture-Dialogue reveals a substantial gap in the current state-of-the-art for Arabic NLP. While proprietary models show promise, open-weight models require significant architectural and training advancements to handle the complexity of culturally grounded, dialectal Arabic dialogue.