Technical Summary: FFE-HALLU

Problem Definition

Large Language Models (LLMs) have demonstrated remarkable success in various NLP tasks but continue to struggle with figurative language, specifically Fixed Figurative Expressions (FFEs) such as idioms and proverbs. Unlike literal phrases, FFEs are culturally grounded, largely non-compositional, and conventionally fixed. The paper identifies a specific, underexplored failure mode termed figurative hallucination: the generation or endorsement of expressions that sound idiomatic and plausible but do not exist as authentic fixed expressions in the target language.

While factual hallucinations can often be verified against external knowledge sources, figurative hallucinations are more insidious due to the "illusory truth effect," where familiar-sounding structures are perceived as true even by informed readers. This poses significant risks for applications requiring cultural competence, such as machine translation and dialogue systems, particularly for non-native users who may accept fabricated expressions as authentic.

Methodology

To address this gap, the authors introduce FFE-HALLU, the first comprehensive benchmark designed to evaluate figurative hallucination in LLMs, with a specific focus on the Persian language. The benchmark consists of 600 curated instances divided into three complementary tasks:

1. FFE Generation from Meaning:

   • Goal: Test if a model can retrieve a valid idiom or proverb given a figurative definition.
   • Data: 200 authentic Persian idioms/proverbs with definitions sourced from the Abadis dictionary and GPT-4o paraphrases.
   • Evaluation: Outputs are categorized as Correct (authentic FFE matching meaning), Incorrect (authentic FFE with mismatched meaning), or Hallucinated (literal phrase or non-existent expression).

2. Fake FFE Detection:

   • Goal: Assess the model's ability to distinguish authentic expressions from plausible fabrications.
   • Data: 200 fabricated FFEs constructed using four controlled strategies:
     • Word Perturbation: Replacing key lexical items in real idioms.
     • Semantic Inversion: Reversing or distorting the meaning of real structures.
     • Structural/Syntactic Mimicry: Generating new expressions following common proverb templates without using real lexical material.
     • Cultural/Historical Fabrication: Using culturally salient elements (e.g., historical figures, rituals) to create convincing but invalid expressions.
   • Evaluation: Models are prompted with "Is this FFE fake?" and "Is this FFE real?" to measure consistency and hallucination rates (false acceptances).

3. FFE-to-FFE Translation (English → Persian):

   • Goal: Evaluate cross-lingual transfer of idiomatic meaning without resorting to literal translation or invention.
   • Data: 200 bilingual English-Persian pairs with verified equivalents.
   • Evaluation: Outputs are categorized as Correct (culturally equivalent FFE), Incorrect (existing FFE with wrong meaning), or Hallucinated (literal translation or invented expression).

Experimental Setup:
The authors evaluated six state-of-the-art multilingual LLMs (GPT-4.1, Claude 3.7 Sonnet, Gemma3-27B, DeepSeek-V3, DeepSeek-R1, and Qwen3-235B). Evaluation involved both human annotation by native Persian speakers and automatic evaluation using LLM-as-a-judge (GPT-4.1 and Gemini 2.5 Pro) to assess the reliability of automated metrics for figurative tasks.

Key Results

• Model Performance Variance: GPT-4.1 demonstrated the most robust performance across all tasks, achieving the highest accuracy in generation (60.5%) and translation (65%), and the lowest hallucination rates. Claude 3.7 Sonnet showed strong generation capabilities but struggled significantly with detecting fake FFEs (61.75% hallucination rate).
• Weakness in Open-Weight Models: Models such as DeepSeek-V3, Gemma3-27B, and Qwen3-235B exhibited severe weaknesses. Qwen3-235B, for instance, hallucinated in nearly 70% of generation tasks and 94% of translation tasks, often producing ungrammatical or entirely fabricated expressions.
• Detection Instability: Performance in the Fake FFE Detection task was highly sensitive to prompt framing. While most models performed well when asked "Is this real?", their accuracy dropped sharply when asked "Is this fake?", indicating instability in rejecting fabricated content.
• Automatic Evaluation Limitations: The study found that automatic evaluation (LLM-as-a-judge) is challenging for figurative hallucination. Gemini 2.5 Pro showed better alignment with human judgments (higher Cohen's κ) than GPT-4.1, particularly in distinguishing hallucinated FFEs from merely incorrect ones. However, automatic judges struggled as hallucination rates increased, often misclassifying non-existent expressions as merely "incorrect" real expressions.

Significance and Contributions

The paper makes four primary contributions:

1. Introduction of FFE-HALLU: The first benchmark specifically designed to evaluate figurative hallucination in LLMs, focusing on fixed figurative expressions in Persian.
2. Task Framework: The proposal of three complementary tasks (Generation, Detection, Translation) that probe distinct aspects of figurative competence and cultural grounding.
3. Empirical Analysis: A detailed evaluation of six SOTA multilingual LLMs, revealing systematic weaknesses in figurative generation, detection, and cross-lingual transfer, particularly in open-weight models.
4. Evaluation Validity: An assessment of the reliability of LLM-as-a-judge methods for figurative hallucination, highlighting the necessity of expert human annotation for culturally nuanced tasks.

The authors conclude that mastery of figurative language requires more than multilingual fluency; it demands genuine cultural understanding. The findings underscore the need for targeted benchmarks to assess and mitigate figurative hallucination, as current models frequently generate or endorse culturally invalid expressions that can mislead users.

Limitations

The authors acknowledge that detecting figurative hallucinations relies heavily on the intuition and judgment of native speakers, which restricted the benchmark to Persian. Additionally, the dataset size (600 items) is modest due to the difficulty of curating and validating idioms at scale. The paper notes that while automatic judging offers scalability, it remains imperfect and should not replace expert human annotation, especially for culturally subtle cases.