Technical Summary: MasalBench

Problem Statement

While Large Language Models (LLMs) have demonstrated proficiency in understanding figurative language within high-resource languages like English, their capabilities in low-resource languages remain underexplored. This gap is particularly critical for Persian, a language rich in proverb usage yet lacking comprehensive benchmarks for evaluating LLMs' contextual and cross-cultural comprehension. Existing studies on Persian proverbs have been limited in scope, often relying on binary-choice formats with simplistic distractors or focusing on small, non-systematic datasets. Consequently, there is a lack of rigorous evaluation regarding how well multilingual LLMs can interpret proverbs within conversational contexts and identify culturally equivalent proverbs across languages.

Methodology

The authors introduce MasalBench, a comprehensive benchmark designed to assess LLMs on two distinct levels: Contextual Understanding and Cross-Cultural Understanding.

Dataset Construction

The benchmark was constructed using Foote Koozegari: Persian Proverbs and Their Stories, a reference containing over 4,000 proverbs. The authors utilized Gemini 2.5 Pro to assist in extracting and processing data from image-based PDFs, followed by rigorous manual verification by native Persian speakers to ensure cultural accuracy and naturalness.

1. Contextual Understanding (1,000 items):

   • Format: Multiple-choice questions (MCQs) based on short, natural dialogues.
   • Structure: In each dialogue, a second speaker responds to a first speaker using a specific Persian proverb. The model must identify the speaker's intent.
   • Distractors: Three incorrect options are generated for each question, categorized specifically to test different failure modes:
     • Literal Trap: A misinterpretation based on the literal meaning of the words.
     • Plausible: A logically coherent but contextually incorrect interpretation.
     • Irrelevant: An option unrelated to the dialogue or proverb.
   • Process: 100 dialogues were manually crafted, while the remainder were generated via Gemini 2.5 Pro and manually verified.

2. Cross-Cultural Understanding (700 items):

   • Format: Binary-choice questions.
   • Task: Given a Persian proverb, the model must identify the correct English equivalent from two options.
   • Distractors: The incorrect option is an "irrelevant" English proverb that shares similar wording, tone, or imagery with the Persian proverb but conveys a different meaning, serving as a convincing distractor.
   • Process: Generated via Gemini 2.5 Pro and manually verified to ensure the equivalents are authentic and the distractors are culturally plausible yet semantically distinct.

Experimental Setup

The benchmark was evaluated on eight state-of-the-art multilingual LLMs, including Llama 4 Scout, Llama 3.3 70B Instruct, Qwen 2.5 72B Instruct, Qwen QwQ 32B, DeepSeek V3.1, DeepSeek R1, GPT-4.1 mini, and GPT-4o mini.

• Prompting: Zero-shot prompting was employed with randomized option orders to prevent positional bias.
• Parameters: Temperature was set to 0 for deterministic outputs, and top-p to 1. Models were restricted to returning only the option letter.

Key Results

Contextual Understanding

Models demonstrated strong performance in identifying the intended meaning of Persian proverbs within dialogue.

• Accuracy: All evaluated models achieved accuracy scores above 0.90.
• Top Performer: DeepSeek V3.1 achieved the highest accuracy (0.943), attributed partly to its large parameter count (671B).
• Error Analysis: When models failed, the errors were predominantly plausible distractors (ranging from 0.06 to 0.08 proportion of errors), indicating that models struggle to distinguish between semantically coherent but contextually incorrect interpretations. Literal and irrelevant errors were rare (<0.02).
• Model Characteristics: Instruction-tuned models generally outperformed their reasoning-focused counterparts within the same families, suggesting that instruction tuning aids in cultural contextualization.

Cross-Cultural Understanding

Performance dropped significantly when models were tasked with mapping Persian proverbs to their English equivalents.

• Accuracy: Scores ranged from 0.647 (Llama 4 Scout) to 0.793 (DeepSeek R1).
• Challenges: The task proved more difficult than contextual understanding, requiring not just linguistic decoding but also cultural knowledge and analogical reasoning to navigate differences in imagery and wording between languages.
• Instruction Tuning: While instruction tuning provided a marginal benefit, its impact was limited compared to the contextual task, as success relied heavily on pre-training exposure to cultural data rather than instruction following.

Significance and Contributions

The paper claims several key contributions to the field of LLM evaluation:

1. MasalBench: The introduction of the first comprehensive benchmark specifically designed to evaluate the contextual and cross-cultural understanding of Persian proverbs, a low-resource language with high cultural reliance on figurative speech.
2. Granular Error Analysis: The study provides a detailed breakdown of model failures, distinguishing between literal, plausible, and irrelevant errors, revealing that current models are robust in metaphorical reasoning but prone to selecting semantically coherent but incorrect interpretations.
3. Cross-Cultural Limitations: The findings highlight a significant gap in current LLM capabilities regarding cross-cultural abstraction. While models can understand proverbs within their native cultural context, transferring this understanding to an equivalent in another language remains a substantial challenge, exposing limitations in cultural grounding and analogical reasoning.
4. Framework for Future Research: MasalBench offers a framework for assessing cross-cultural understanding in other low-resource languages, addressing a critical gap in the evaluation of multilingual LLMs.

The authors conclude that while multilingual LLMs are becoming robust in intra-language metaphorical reasoning, they require further development to achieve true cross-cultural fluency, particularly in tasks demanding cultural equivalence rather than direct translation.