A Parallel Cross-Lingual Benchmark for Multimodal Idiomaticity Understanding

1. Problem Statement

Idioms and Potentially Idiomatic Expressions (PIEs) pose significant challenges for Natural Language Processing (NLP) systems due to their non-compositional nature, deep reliance on cultural knowledge, and resistance to literal translation. While Large Language Models (LLMs) have advanced in general linguistic tasks, they struggle with idiomatic understanding, often failing to distinguish between literal and figurative meanings.

Existing benchmarks suffer from three main limitations:

1. Lack of Parallelism: Most multilingual idiom datasets are not parallel; idioms are not systematically aligned across languages, preventing direct cross-linguistic comparison of how specific concepts are realized.
2. Monomodal Focus: Most datasets rely solely on text, ignoring the multimodal nature of language where visual context can aid or hinder understanding.
3. Limited Scope: Many existing resources focus on high-resource European languages, neglecting endangered or low-resource languages and the diversity of global idiomatic expressions.

2. Methodology

The authors present XMPIE, a parallel, multilingual, and multimodal dataset designed to bridge these gaps.

Dataset Construction

• Scale and Coverage: The dataset covers 34 language variants (ranging from widely spoken languages like English and Chinese to endangered ones like Aromanian and Luxembourgish) and contains 3,054 PIEs derived from English seed expressions. In total, it includes over 10,000 items (including literal/figurative variations and distractors) and 7,040 images.
• Annotation Process:
  • Human-in-the-Loop: 89 native or highly fluent language experts were recruited via the UNIDIVE network.
  • Translation & Mapping: Experts provided: (1) literal translations, (2) idiomatic equivalents, (3) transliterations, and (4) back-translations to English.
  • Image Generation: Using Midjourney, experts generated five images per PIE representing a spectrum of meaning:
    1. Idiomatic sense (Target 1)
    2. Idiomatic-related (Weak variant)
    3. Literal-related (Weak variant)
    4. Literal sense (Target 2)
    5. Random distractor (Unrelated)
  • Quality Control: Prompts were carefully crafted to be visually grounded, avoiding abstract concepts that image models struggle with. Images were saved in high resolution and vetted for safety and relevance.

Cross-Lingual Alignment Analysis

The authors analyzed the mapping between English seed PIEs and their target language equivalents, categorizing them into four alignment types:

• 1-1: Direct idiomatic equivalent (identical or near-identical wording/structure).
• 1-0: Meaning conveyed via descriptive (non-idiomatic) phrasing or single words.
• 1-N: One English idiom maps to multiple target idioms (reflecting ambiguity).
• N-1: Multiple English idioms map to a single target idiom.

Graph-Based Representation

To quantify cross-linguistic similarity, the authors modeled each PIE as a graph where nodes represent language-specific equivalents (via literal translation) and edges represent lexical overlap. Metrics such as graph density, connected components, and clique numbers were computed to measure how "convergent" or "divergent" an idiom is across languages.

3. Key Contributions

1. XMPIE Dataset: The first large-scale, parallel multilingual and multimodal benchmark for idiomaticity, covering 34 languages and offering a unique view into cultural and linguistic diversity.
2. Multimodal Evaluation Protocol: A novel framework for evaluating how models interpret idioms across text and vision modalities, specifically testing the ability to distinguish between literal and figurative visual representations.
3. Cross-Lingual Insights: A detailed analysis of idiom mapping types (1-1, 1-0, etc.), revealing that while some concepts (e.g., "black market") are globally convergent, others (e.g., "acid test") are highly culture-specific and fragmented.
4. Baseline Performance: Establishment of a rigorous baseline using state-of-the-art vision-language models (EVA-CLIP-18B) to highlight current limitations in idiomatic understanding.

4. Results

The authors evaluated the EVA-CLIP-18B model (a retrieval-style baseline) on a subset of five languages (English, Brazilian Portuguese, Spanish, Chinese, Turkish) without fine-tuning.

• High NDCG, Low Precision: The model achieved high NDCG@5 scores (0.87–0.95), indicating it generally ranks relevant images (literal, idiomatic, and weak variants) near the top.
• Literal Bias: There is a stark asymmetry in performance. The model consistently identifies the literal image with high accuracy (e.g., English Top-1 Literal: 0.900) but fails significantly on the idiomatic image (e.g., English Top-1 Idiomatic: 0.060).
• Strict Ranking Failure: Under the stricter Top-2 metric (requiring both literal and idiomatic images to be in the top two), idiomatic performance drops to near zero (e.g., English Top-2 Idiomatic: 0.010).
• Language Variation: While most languages showed a literal bias, Spanish (ES) was an exception where idiomatic ranking outperformed literal ranking, suggesting language-specific nuances in how visual models process these concepts.
• Conclusion on Baseline: Current vision-language models struggle to disambiguate idiomatic meaning from literal meaning without contextual cues, relying heavily on visual literalness.

5. Significance

• Cultural Benchmarking: XMPIE moves beyond linguistic competence to assess cultural sensitivity in AI, crucial for developing models that understand diverse global perspectives.
• Resource for Low-Resource Languages: By including endangered and low-resource languages, the dataset supports research into how idiomaticity is handled in data-scarce scenarios.
• Multimodal Understanding: The dataset highlights the gap between text-based and vision-based understanding of figurative language, providing a testbed for improving multimodal reasoning.
• Future Directions: The paper underscores the need for models to utilize contextual cues (beyond just the phrase itself) to achieve robust idiomatic understanding and suggests future work involving abstractness and imageability factors.

In summary, XMPIE provides a critical infrastructure for evaluating and improving the "cultural intelligence" of NLP and multimodal models, revealing that while models can retrieve relevant concepts, they currently lack the deep semantic and cultural grounding required to truly understand idioms across languages and modalities.