Unmasking the Factual-Conceptual Gap in Persian Language Models

1. Problem Statement

Current Large Language Model (LLM) benchmarks for Persian primarily focus on linguistic competence, factual retrieval, and translation. However, they fail to evaluate cultural reasoning, specifically the ability to distinguish between memorized cultural facts and the application of implicit social schemas.

Persian culture is characterized by complex, context-dependent norms (e.g., Taarof, superstitions, and rituals) that often lack logical justification and rely on historical syncretism (Zoroastrian, Islamic, and folk beliefs). The core problem is that current models may mimic surface-level cultural patterns (pattern-matching) without internalizing the underlying logic required to navigate social situations, leading to a gap between cultural fluency (speaking correctly) and cultural competence (acting appropriately).

2. Methodology: DIVANBENCH

The authors introduce DIVANBENCH, a diagnostic benchmark designed to isolate and measure cultural reasoning capabilities.

• Scope: Covers 81 distinct cultural concepts across 7 domains (Social Etiquette, Nowruz Traditions, Supernatural Beings, Apotropaic Rituals, Wedding Ceremonies, Taboos, and Divination).
• Dataset Size: 315 questions.
• Task Architecture: The benchmark employs a three-tiered evaluation structure:
  1. Factual MCQ (100 questions): Tests baseline knowledge retrieval (e.g., "What is Taarof?").
  2. Binary Belief Verification (162 statements): Uses a paired design for 81 concepts. Each concept has a "positive" scenario (adhering to norms) and a "negative" scenario (violating norms). This specifically measures acquiescence bias (the tendency to agree with plausible-sounding statements regardless of truth).
  3. Scenario-Based MCQ (53 questions): Tests multi-step inference in novel, complex social contexts requiring the application of cultural logic rather than rote memory.
• Evaluation Protocol:
  • Evaluated 7 state-of-the-art Persian-capable LLMs (7B–12B parameters), including Aya, Dorna2, Gemma2/3, Llama3.1, and Qwen2/2.5.
  • Key Comparison: A controlled comparison between Llama3.1-8B (base) and Dorna2-8B (Persian-adapted via continuous pretraining) to isolate the effect of monolingual pretraining.
  • Metrics: Accuracy, Acquiescence Bias (Acceptance Rate of Positive - Rejection Rate of Negative), and Factual-Conceptual Gap.

3. Key Contributions

1. DIVANBENCH: The first benchmark specifically targeting the distinction between factual cultural knowledge and the reasoning required for implicit social norms in Persian.
2. Bias Quantification: A novel methodology using paired positive/negative scenarios to explicitly quantify acquiescence bias (the "Yes-saying" phenomenon) in cultural contexts.
3. Pretraining Analysis: A direct empirical study on how continuous monolingual pretraining affects reasoning versus fluency, challenging the assumption that more target-language data always improves understanding.
4. Theoretical Framework: Provides evidence supporting the cognitive anthropology distinction between Cultural Facts (retrievable from text) and Cultural Schemas (requiring embodied, context-dependent learning).

4. Key Results

The evaluation of 7 models yielded four critical findings:

• The Acquiescence Trap:

  • 6 out of 7 models exhibited severe acquiescence bias. While they correctly identified appropriate behaviors (84–92% accuracy), they failed to reject clear cultural violations (19–48% rejection rate).
  • Example: Models accepted both "refusing food twice then accepting" (correct Taarof) and "accepting immediately" (violation) as plausible, indicating reliance on keyword matching rather than logical deduction.
  • Exception: Llama3.1-8B showed a "skeptical bias" (73% rejection), suggesting it was less prone to pattern-matching than its Persian-adapted counterpart.

• The Persian Pretraining Paradox:

  • Continuous pretraining on Persian data degraded critical reasoning.
  • Dorna2-8B (Persian-adapted) saw its rejection accuracy drop from 73% (Llama3.1) to 30%, a 43-percentage-point decline.
  • While factual knowledge remained relatively stable, the model became "culturally compliant" but less discerning, accepting any culturally themed statement regardless of correctness.

• The Factual-Conceptual Gap:

  • There is a consistent 21% performance gap between Factual MCQs and Scenario-Based reasoning across all models.
  • Models can retrieve facts (e.g., "What is Esfand?") but fail to apply the logic in new scenarios (e.g., "Is burning Esfand appropriate in this specific context?"). This suggests memorization does not equate to schema internalization.

• Size Does Not Guarantee Intelligence:

  • The largest model, Gemma3-12B, achieved the highest factual accuracy (87%) but exhibited extreme acquiescence bias (+61%), performing worse on reasoning than smaller models. This indicates that scaling parameters improves fact memorization but may amplify pattern-matching behaviors without enhancing cultural discernment.

5. Significance and Implications

• Rethinking Low-Resource NLP: The study challenges the prevailing assumption that scaling monolingual data for low-resource languages (like Persian) automatically leads to better cultural understanding. Instead, it suggests that without specific reasoning training, more data reinforces surface-level patterns and biases.
• Beyond Fluency: Cultural competence requires more than linguistic fluency; it demands the ability to navigate implicit, context-dependent social contracts. Current training objectives (Next Token Prediction) are insufficient for learning these schemas.
• Future Directions: The findings suggest that improving cultural reasoning in LLMs requires training mechanisms beyond simple data scaling, such as explicit schema learning, reinforcement learning from human feedback (RLHF) focused on cultural logic, or synthetic data generation that emphasizes counter-factual reasoning.
• Open Resource: The authors have released DIVANBENCH publicly to serve as a template for evaluating cultural reasoning in other languages, moving beyond standard NLP benchmarks.

In conclusion, the paper demonstrates that current Persian LLMs are "cultural mimics" rather than "cultural reasoners," capable of reciting facts but failing to understand the nuanced, often illogical, social rules that govern human interaction in Persian society.