When Meaning Isn't Literal: Exploring Idiomatic Meaning Across Languages and Modalities

1. Problem Statement

Current Large Language Models (LLMs) and Vision Language Models (VLMs) struggle with idiomatic reasoning, particularly in low-resource languages and multimodal contexts.

• The Core Issue: Idioms rely on cultural context and metaphorical meaning rather than literal word definitions. Models often fail to distinguish between literal and figurative interpretations, leading to "naïve" reasoning (e.g., interpreting the Bengali idiom Angur fol tok literally as "grapes are sour" rather than understanding the metaphor of "sour grapes" or denial-driven rationalization).
• The Gap: Existing research and datasets are heavily English-centric. There is a lack of high-quality, multimodal resources for Hindi, Bengali, and Thai that pair text with culturally aligned images and gold-standard explanations. Furthermore, current models lack mechanisms to iteratively correct their reasoning errors regarding cultural nuances.

2. Methodology

The authors propose a two-pronged approach: the creation of a new dataset (Mediom) and a novel reasoning framework (HIDE).

A. Mediom: A Multimodal, Multilingual Idiom Corpus

• Scale & Scope: The dataset contains 3,533 idioms across Hindi, Bengali, and Thai.
• Data Composition: Each entry includes:
  • The original idiom text.
  • Gold-standard explanations: Literal translation, contextual interpretation, usage scenarios, cultural significance, and coherence checks.
  • Cross-lingual translations.
  • Aligned Text-Image Representations: High-fidelity images generated via a prompt-driven pipeline (using DALL·E 3 and GPT-4o) to ensure the visuals capture the metaphorical meaning rather than just literal depictions.
• Quality Assurance: Annotations were performed by native-speaking experts (professors and scholars) using an Information Persistence Rating (IPR) criteria. A two-tier validation (peer review and expert panel) ensured a high inter-annotator agreement (Cohen's kappa = 0.82).

B. HIDE: Hinting-based Idiom Explanation Framework

To address the reasoning gaps, the authors propose HIDE, a framework inspired by Error Feedback Learning (EFL).

• Mechanism:
  1. Error Detection: The model generates an initial interpretation of an idiom.
  2. Feedback Loop: If the interpretation is incorrect, the system compares the model's output against the gold-standard annotation.
  3. Hint Generation: A discriminator compresses the discrepancy into a high-level corrective hint (e.g., "Consider whether the scene represents a broader emotion rather than a literal description").
  4. Retrieval & Refinement: During inference, the system retrieves semantically similar past errors from an Idiomatic Error-Feedback Repository. The retrieved hint is concatenated with the input prompt to guide the model toward the correct figurative meaning iteratively.
• Models Tested: The framework was evaluated on various LLMs (Gemma2-9B, Mistral-7B, LLaMA2-7B) and VLMs (Paligemma2-10B, Qwen2-VL, etc.).

3. Key Contributions

1. Mediom Dataset: The first multimodal, cross-cultural idiom benchmark for Hindi, Bengali, and Thai, featuring 3,533 rigorously annotated idioms with paired text and metaphorically accurate images.
2. HIDE Framework: A novel, hint-driven reasoning loop that leverages error feedback to refine model outputs, specifically targeting cultural and metaphorical comprehension.
3. Comprehensive Benchmarking: A dual evaluation of textual reasoning (LLMs) and figurative disambiguation (VLMs), revealing systematic failures in current state-of-the-art models.

4. Experimental Results

The study evaluated models on the Mediom dataset using metrics like ROUGE, BLEU, BERTScore, and perplexity.

• Baseline Performance (RQ1):
  • Text-only LLMs significantly outperformed VLMs in idiomatic understanding. For instance, Gemma2-9B achieved the highest scores (ROUGE-1: 0.65), while the best VLM (Paligemma2-10B) scored lower (ROUGE-1: 0.43).
  • This indicates that large-scale textual pretraining is currently more effective at internalizing cultural semantics than current multimodal grounding.
• Impact of HIDE (RQ2):
  • Integrating HIDE yielded consistent improvements across all models.
  • Gemma2-9B + HIDE showed significant gains: ROUGE-1 increased to 0.68, Cosine Distance dropped to 0.21, and Perplexity decreased, indicating sharper semantic alignment.
  • The framework successfully converted past errors into "micro-lessons," helping models escape literal traps.
• Generalizability (RQ3):
  • The dataset proved robust across different model families.
  • Qualitative Analysis: HIDE-equipped LLMs demonstrated superior ability to capture cultural metaphors (e.g., understanding "blaming the courtyard" as a metaphor for shifting blame). VLMs showed improvement in visual grounding but still struggled with deep figurative abstraction compared to text-only models.

5. Significance and Conclusion

• Cultural AI: The work highlights that "blind spots" in AI are often cultural. By focusing on low-resource languages, the authors demonstrate that idiomatic competence requires more than just lexical knowledge; it demands cultural fluency.
• Methodological Shift: The HIDE framework offers a new paradigm for improving reasoning in AI. Instead of just retraining models, it suggests using retrieved error feedback to provide targeted, iterative hints during inference.
• Future Directions: The paper establishes a rigorous testbed for developing next-generation AI systems that are culturally grounded and multimodal. It suggests that while VLMs are improving, the integration of explicit reasoning hints (like HIDE) is crucial for bridging the gap between literal perception and figurative understanding.

In summary, the paper argues that to achieve human-level idiomatic reasoning, AI systems must move beyond surface-level cues and utilize structured, culturally-aware feedback loops like HIDE, supported by high-quality, multimodal datasets like Mediom.