💬 NLP

Beyond Understanding: Evaluating the Pragmatic Gap in LLMs' Cultural Processing of Figurative Language

This paper evaluates the pragmatic gap in large language models' cultural processing by demonstrating that while they can interpret figurative language, they struggle significantly more with culturally grounded expressions like Egyptian Arabic idioms and proverbs compared to English, prompting the release of the Kinayat dataset to support future research.

Original authors: Mena Attia, Aashiq Muhamed, Mai Alkhamissi, Thamar Solorio, Mona Diab

Published 2026-02-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mena Attia, Aashiq Muhamed, Mai Alkhamissi, Thamar Solorio, Mona Diab

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant new student named LLM (Large Language Model). This student has read almost every book, website, and article on the internet. They can recite facts, write poems, and solve math problems better than almost anyone.

But there's a catch: LLM is like a tourist who has memorized a phrasebook but hasn't actually lived in the country. They know the words, but they don't quite get the vibe, the inside jokes, or the unwritten rules of how people actually talk to each other.

This paper is like a "cultural pop quiz" designed to see if our super-smart student can actually get the local culture, or if they are just faking it.

The Test: Idioms and Proverbs

The researchers didn't ask LLM to solve calculus. Instead, they tested it on idioms and proverbs.

Think of an idiom like a secret handshake.

Literal meaning: "Don't sell water in the village of water-sellers."
Real meaning: "Don't try to sell your wares to people who already have plenty of them" (or, "Don't try to outsmart the experts").

If you just translate the words, it makes no sense. You need to understand the story behind it. The paper tested LLMs on:

English Proverbs: The "textbook" examples.
Arabic Proverbs: The "standard" cultural examples.
Egyptian Arabic Idioms: The "street smart," slang-heavy, local jokes.

The Results: The "Tourist" vs. The "Local"

The results revealed a clear hierarchy, like a video game with increasing difficulty levels:

Level 1: English Proverbs (The Easy Mode)
LLMs were pretty good here. They could explain the meaning of English sayings because they had read so many English books. They got about 90% right.
Level 2: Arabic Proverbs (The Medium Mode)
When they switched to formal Arabic proverbs, the scores dropped by about 4%. The models were still okay, but they started to stumble a bit on the cultural nuances.
Level 3: Egyptian Idioms (The Hard Mode)
This is where it got messy. When the models tried to understand Egyptian slang and idioms (which are very specific to daily life, work, and family dynamics), their scores dropped another 10%. They were essentially guessing.

The Big Surprise: The "Pragmatic Gap"

Here is the most interesting part of the story. The researchers gave the models two types of tests:

Test A (The Quiz): "Here is an idiom. What does it mean?"
- Result: The models were decent at this. They could pick the right definition from a list.
Test B (The Party): "Here is a situation. Use an idiom to describe it naturally."
- Result: Total failure.

The Analogy:
Imagine you are at a party. You know the definition of the word "awkward."

Test A: If someone asks, "What does 'awkward' mean?" you can say, "It means uncomfortable." (You pass).
Test B: If someone drops a tray of glasses, and you immediately shout, "That is a classic example of awkward!" while everyone is staring at you, you have failed the party. You know the word, but you don't know when or how to use it.

The paper found that LLMs are terrible at Test B. Even the smartest models dropped their accuracy by 14% when asked to use the idiom correctly in a sentence, compared to just explaining it. They sound like a robot reading a dictionary, not a human having a conversation.

The "Cultural Blind Spot"

The study also looked at connotations (the emotional feeling of a word).

Is a phrase funny? Is it insulting? Is it sarcastic?
Even when humans agreed 100% on the meaning, the models only agreed with them about 85% of the time.

It's like the models are colorblind to the "emotional colors" of language. They see the black and white text, but they miss the rainbow of feelings behind it.

The New Tool: "Kinayat"

To help fix this, the authors created a new dataset called Kinayat.
Think of this as a new "Cultural Survival Guide" for AI. It's a collection of 325 Egyptian idioms, carefully labeled with their meanings and examples of how to use them in real life. It's like giving the tourist a map that doesn't just show the streets, but also shows where the good coffee shops are and which streets to avoid.

The Bottom Line

This paper teaches us that knowing a language is not the same as understanding a culture.

Current AI models are like encyclopedias that can recite facts perfectly. But when it comes to the messy, emotional, cultural, and "inside joke" parts of human conversation, they are still just tourists. They can read the menu, but they haven't learned how to order the food like a local yet.

The takeaway: If you want an AI to truly understand your culture, you can't just feed it more text. You have to teach it the context, the feelings, and the unwritten rules of how people actually talk.

1. Problem Statement

While Large Language Models (LLMs) have demonstrated significant progress in multilingual text generation and literal comprehension, their ability to process culturally grounded meaning remains under-explored. Specifically, LLMs struggle with figurative language (idioms, proverbs, metaphors), which encodes local knowledge, social values, and pragmatic nuances that extend beyond surface-level semantics.

Current benchmarks primarily focus on figurative comprehension (e.g., explaining the meaning of an idiom) but neglect pragmatic use (correctly applying an idiom in a specific social context) and connotative inference (understanding the emotional or cultural weight of an expression). This gap limits the assessment of true cultural competence in AI systems.

2. Methodology

The authors propose a comprehensive evaluation framework designed to test LLMs across three dimensions: interpretation, pragmatic use, and connotation.

Datasets

Kinayat (New Contribution): A novel dataset of 325 Egyptian Arabic colloquial idioms (extracted from Al-Kinayat Al-'Amiyya, 1949), annotated with Modern Standard Arabic (MSA) explanations. A subset of 150 idioms was curated for pragmatic use evaluation, with sentences generated and verified by native speakers.
Jawaher: A dataset of 198 multidialectal Arabic proverbs across 20 dialects.
MAPS: A multilingual dataset of proverbs; the authors utilized the English test set (394 proverbs) for cross-lingual comparison.

Evaluation Tasks

The study evaluates 22 models (open-source and closed-source) across the following tasks:

Multiple Choice Question (MCQ) Understanding: Models select the correct figurative meaning from two options (one correct, one incorrect).
- Distractor Generation: Incorrect options are generated using two strategies: a general plausibility prompt and a Semantic Role Labeling (SRL) approach to create subtly incorrect explanations.
Pragmatic Use: Models must select the correct idiom to fill a blank in a given sentence context. This tests the ability to apply figurative language appropriately.
Contextual Understanding: An extension of the MCQ task where the model is provided with an idiomatic sentence as context before selecting the meaning.
Negation: Models are asked to identify the incorrect explanation, testing their ability to reject valid options.
Proverb Completion: Models complete a proverb by predicting the final word (testing memorization).
Connotation Labeling: Models classify idioms/proverbs as Positive, Negative, or Neutral. Evaluation is restricted to samples with 100% human inter-annotator agreement to minimize subjectivity.

Models Evaluated

22 Models: Including LLaMA 3.1, Gemma 2, Qwen 2.5, Aya-Expanse, Mistral, Jais, ALLaM, SILMA, Fanar, AceGPT, and closed-source models (GPT-4o, Claude 3.5/4, Gemini 1.5/2.5).
Evaluation Metrics: Accuracy for MCQ tasks; BERTScore and LLM-as-a-Judge for generation tasks; Human verification for distractor quality.

3. Key Contributions

Pragmatic Use Task: Introduction of a novel evaluation task assessing the ability of LLMs to use figurative language in context, revealing a significant gap between understanding and application.
Kinayat Dataset: Release of the first dataset specifically designed for Egyptian Arabic idioms, annotated for both figurative meaning and pragmatic use, addressing a gap in Arabic cultural benchmarks.
Unified Cross-Lingual Framework: A comprehensive suite evaluating interpretation, contextual appropriateness, and connotative inference across English, MSA, and Egyptian dialects.

4. Key Results

Performance Hierarchy

A consistent performance degradation was observed across languages and dialects:

English Proverbs (MAPS): Highest performance (Avg. ~90.86% without context, ~95.66% with context).
Arabic Proverbs (Jawaher): ~4.29% lower than English.
Egyptian Idioms (Kinayat): ~10.28% lower than Arabic proverbs (Avg. ~76.29%).
Insight: Models struggle most with colloquial dialects and idioms, likely due to lower representation in training data compared to MSA and English.

The "Pragmatics Gap"

Understanding vs. Use: There is a significant drop in performance when moving from understanding to pragmatic use.
- Average accuracy on Understanding: ~78.52%.
- Average accuracy on Pragmatic Use: ~64.45%.
- Gap: A 14.07% decrease in accuracy.
Contextual Aid: Providing the idiomatic sentence as context in the MCQ task improved accuracy by 10.66%, suggesting models rely heavily on surface cues rather than deep cultural reasoning.

Connotation and Negation

Connotation: Models struggle with affective dimensions. Even on samples with 100% human agreement, the best models (Claude 3.5) reached only 85.58% accuracy on idioms.
Negation: Performance dropped significantly when models were asked to identify the incorrect explanation (e.g., from 76.29% to 70.97% for idioms), indicating difficulty in rejecting valid options.

Model Size and Architecture

Scale Correlation: A weak positive correlation exists between model size and performance ( $R^2 \approx 0.26$ for understanding, $R^2 \approx 0.60$ for pragmatic use). Larger models generally perform better, particularly in pragmatic reasoning.
Closed-Source Dominance: Closed-source models (Claude, GPT-4, Gemini) consistently outperformed open-source models, likely due to larger parameter counts and more diverse training data.
Arabic Models: While specialized Arabic models (e.g., Jais, ALLaM) showed strengths in completion tasks, they generally lagged behind top-tier multilingual/closed-source models in pragmatic use.

5. Significance and Implications

Cultural Reasoning Diagnostic: The study establishes figurative language as a robust diagnostic tool for cultural reasoning. The inability of models to use idioms pragmatically suggests they lack the "cultural substrate" necessary for true social intelligence.
Limitations of Current LLMs: The results demonstrate that while LLMs can often interpret figurative meaning (surface semantics), they fail to apply it appropriately in social contexts (pragmatics). This indicates a reliance on statistical patterns rather than deep cultural internalization.
Future Directions: The paper calls for benchmarks that extend beyond static comprehension to dynamic, context-aware generation and evaluation across diverse dialects and cultural contexts. The release of Kinayat provides a foundational resource for this next generation of cultural AI research.

In conclusion, the paper highlights a critical "Pragmatics Gap" in current LLMs, proving that high accuracy in literal or explanatory tasks does not equate to cultural competence. True fluency requires the ability to navigate the nuanced, context-dependent nature of figurative language.