💬 NLP

Figurative and Cultural Knowledge in LLMs: Investigating Cross-Domain Transfer through Fine-Tuning

This paper investigates whether fine-tuning on cultural data enhances figurative language understanding in LLMs, finding that while poetry training improves idiom comprehension in a transferable manner, cultural fine-tuning often degrades proverb interpretation and reveals a complex, non-linear relationship between cultural immersion and figurative language acquisition that cannot be straightforwardly captured through fine-tuning alone.

Original authors: Mena Attia, Mona Diab, Thamar Solorio

Published 2026-08-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mena Attia, Mona Diab, Thamar Solorio

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Language is more than a set of rules for arranging words; it is a living archive of a community's history, values, and shared experiences. This is especially true for figurative language—the idioms, proverbs, and poetry that rely on cultural context to make sense. A phrase like "it's raining cats and dogs" means nothing to someone who has never heard the expression, but to a native speaker, it instantly conjures a specific image of heavy rain. For artificial intelligence, which learns by processing vast amounts of text, mastering this kind of language is a significant hurdle. While modern AI models can read and write fluently, they often struggle to understand the deeper, non-literal meanings that are woven into the fabric of a culture. The question researchers have long asked is whether teaching an AI about a culture's daily life and traditions would help it understand its metaphors, or if learning to interpret metaphors would help it grasp cultural facts.

A team of researchers set out to test this connection using Arabic, a language rich with diverse dialects and a deep literary tradition. They worked with four different large language models, which are the powerful computer systems that drive many modern AI applications. To see if cultural knowledge and figurative understanding are linked, the researchers conducted a series of controlled experiments. They took these models and gave them extra training, or "fine-tuning," on specific types of data. In one group, they fed the models thousands of examples of cultural knowledge, such as facts about social customs, regional practices, and everyday life across the Arab world. In another group, they trained the models on figurative language, using datasets filled with proverbs, idioms, and poetry. They also included a control group trained on general Arabic grammar to ensure that any changes were due to the specific content of the training and not just the act of learning more Arabic.

The results revealed a complex and somewhat surprising picture. The researchers found that teaching a model about culture did not automatically make it better at understanding idioms or proverbs. In fact, for models that were already specialized in Arabic, training them on cultural data sometimes made their performance worse. These models, which had likely already absorbed a great deal of cultural knowledge during their initial training, seemed to become confused when exposed to a narrower, curated set of cultural facts. Conversely, training models on cultural data did not help them answer general questions about cultural facts either. The two types of knowledge—cultural facts and figurative meaning—did not seem to support each other in the way the researchers had hoped.

However, there was one clear exception that stood out. When the researchers trained the models specifically on poetry, the models became significantly better at understanding idioms. This improvement was measurable and statistically reliable, with a specific model showing a notable jump in accuracy. Crucially, this gain did not happen when the models were trained on general Arabic text or grammar, which suggests that the improvement came from the poetic content itself, not just from seeing more Arabic words. Poetry, with its reliance on imagery, rhythm, and non-literal meaning, appears to have taught the models a broader skill: the ability to recognize and interpret meaning that goes beyond the literal definition of words. This skill transferred successfully to idioms, even though poetry and idioms are different forms of expression.

The study also highlighted that the results depended heavily on which model was being used. The models that were designed specifically for the Arabic language started with a higher baseline of knowledge and often regressed, or performed worse, after being fine-tuned on new data. This suggests they had already learned as much as they could from the available training materials. In contrast, multilingual models, which are trained on many languages, showed more room to learn and improved more consistently when exposed to new data. When the researchers looked closely at the errors, they found that the training tended to reinforce knowledge about everyday, experiential things like food, games, and celebrations, while sometimes weakening the models' grasp of historical facts or specific political details.

Ultimately, the research suggests that the relationship between culture and figurative language is not a simple switch that can be flipped by feeding an AI more data. While the two are deeply connected in human experience, teaching an artificial intelligence one does not guarantee it will master the other. The only reliable bridge found in this study was from poetry to idioms, indicating that exposure to the artistic and non-literal side of a language can help a model navigate its metaphorical landscape. For now, the path to truly culturally aware artificial intelligence remains uneven, requiring more than just a simple transfer of knowledge from one domain to another.

Technical Summary: Figurative and Cultural Knowledge in LLMs

Problem Statement
Figurative language (idioms, proverbs, poetry) is deeply culturally embedded, relying on shared values, social norms, and historical experience rather than linguistic competence alone. While cognitive linguistics and cultural studies establish that figurative competence and cultural knowledge are coupled capacities, Natural Language Processing (NLP) typically treats them as separate domains. Benchmarks for cultural knowledge focus on factual reasoning and commonsense, while figurative benchmarks evaluate non-literal interpretation. It remains untested whether supervision in one domain (e.g., cultural facts) improves performance in the other (e.g., idiom interpretation), or if figurative training enhances general cultural understanding. Furthermore, it is unclear whether observed improvements stem from specific content transfer or merely from increased exposure to the Arabic language.

Methodology
The authors conducted a systematic study using controlled fine-tuning experiments on four Large Language Models (LLMs): two Arabic-centric models (ALLaM-7B, Fanar-1-9B) and two multilingual models (Qwen3-8B, LLaMA-3.1-8B). The study utilized six datasets spanning diverse Arabic dialects and regions:

Cultural Datasets: ArabCulture and Palm (used for cultural fine-tuning).
Figurative Datasets: Jawaher (proverbs), Kinayat (Egyptian idioms), and FannOrFlop (poetry).
Control Dataset: ArabicMMLU (general Arabic language and grammar) to distinguish content-specific transfer from general language adaptation.

The researchers designed three complementary experimental directions:

Cultural $\to$ Figurative: Fine-tuning on cultural data to evaluate impact on idiom and proverb understanding.
Figurative $\to$ Cultural: Fine-tuning on poetry and proverbs to evaluate impact on cultural commonsense reasoning.
Cross-Figurative Transfer: Fine-tuning on one figurative type (e.g., poetry) to evaluate understanding of another (e.g., idioms).

All models were fine-tuned using Low-Rank Adaptation (LoRA) with a 1,000-sample subset (except Jawaher, which had 800 samples). Evaluation was performed via zero-shot inference on held-out test sets, with accuracy measured across three random seeds to account for answer ordering sensitivity. Statistical significance was assessed using paired bootstrap confidence intervals and exact McNemar tests.

Key Results
The study found that cross-domain transfer between cultural and figurative knowledge is generally limited, inconsistent, and highly model-dependent.

Cultural to Figurative: Fine-tuning on cultural datasets (ArabCulture, Palm) did not reliably improve figurative language benchmarks. In fact, cultural fine-tuning caused statistically significant regressions in proverb interpretation for Arabic-centric models (ALLaM and Fanar) when fine-tuned on Palm, suggesting these models had already internalized relevant knowledge during pre-training.
Figurative to Cultural: Fine-tuning on figurative data (proverbs and poetry) produced no detectable improvements on general cultural commonsense benchmarks (AraDiCE).
Cross-Figurative Transfer: The only statistically reliable positive effect observed was poetry fine-tuning improving idiom comprehension. Models fine-tuned on FannOrFlop (poetry) showed a pooled gain of +2.33% on the Kinayat (idiom) benchmark ( $p < 0.05$ ). This gain was not reproduced by the ArabicMMLU control, indicating it stems from the figurative nature of the poetry data rather than general language adaptation.
Model Dependence: Multilingual models (LLaMA, Qwen) generally showed positive point estimates and greater "adaptation headroom," whereas Arabic-centric models (ALLaM, Fanar) frequently regressed, likely due to prior saturation of the relevant cultural and figurative knowledge.
Error Analysis: Fine-tuning tended to reinforce experiential cultural knowledge (e.g., food, traditional games, celebrations) while destabilizing historically grounded or domain-specific factual knowledge (e.g., history, politics, religion).

Significance and Claims
The paper claims that the relationship between cultural and figurative language, while conceptually linked, is not straightforwardly captured through fine-tuning alone. The authors argue that:

Transfer is Limited: Cross-domain transfer between cultural facts and figurative interpretation is not a robust mechanism for improving LLM performance in Arabic.
Specificity of Figurative Transfer: The successful transfer from poetry to idioms suggests that fine-tuning on one figurative form can develop a broader sensitivity to non-literal meaning that generalizes across different figurative types, distinct from general language learning.
Model Saturation: Arabic-centric models may have already absorbed sufficient cultural and figurative knowledge during pre-training, making them less responsive to further fine-tuning on curated, narrower corpora compared to multilingual models.
Knowledge Rebalancing: Fine-tuning can inadvertently shift a model's knowledge profile, favoring culturally embedded, experiential knowledge over factual historical knowledge.

The authors conclude that while dedicated evaluation frameworks are necessary, current Arabic resources and fine-tuning strategies offer only constrained support for bridging the gap between cultural and figurative understanding. They caution against over-interpreting small gains and highlight the need for future work on larger models and different adaptation strategies.

More like this