Technical Summary: MIDI – A Multilingual Idiom Dataset for High-, Medium-, and Low-Resource Languages

1. Problem Statement

Idiomatic expressions present a significant challenge for Multilingual Large Language Models (mLLMs) due to the semantic ambiguity between literal and figurative meanings. While prior research has addressed idiom comprehension, existing benchmarks suffer from critical limitations:

• Language Bias: Most datasets focus on high-resource languages (e.g., English, Chinese), neglecting low-resource languages where training data is sparse and cultural grounding is essential.
• Contextual Deficiency: Previous evaluations often rely on isolated sentences or definition-matching tasks, failing to capture the pragmatic cues found in realistic discourse (e.g., multi-turn dialogues).
• Mechanism Opacity: It remains unclear whether models succeed by memorizing idiom definitions or by reasoning through context to disambiguate usage.

The paper posits that current models struggle to generalize idiom understanding across diverse linguistic landscapes, particularly in low-resource settings and when distinguishing literal from figurative readings.

2. Methodology

Dataset Construction: MIDI

The authors introduce MIDI (Multilingual Idiom Dataset), a comprehensive benchmark curated by native speakers across 18 languages and dialects. The languages are categorized into three resource tiers based on digital availability:

• High-Resource (3): Chinese, Japanese, Russian.
• Medium-Resource (3): Indonesian, Vietnamese, and the UAE dialect of Arabic.
• Low-Resource (12): Javanese, Persian, Kannada, Telugu, Tamil, Minangkabau, Sundanese, Kazakh, Yoruba, and Arabic dialects (Syrian, Egyptian, Moroccan).

Key Features of MIDI:

• Scale: Contains 2,278 idioms with 8,378 usage contexts.
• Dual Contexts: Each idiom is embedded in two realistic discourse settings:
  1. Sentence-level: Crafted in both literal and figurative senses.
  2. Dialogue-level: Short, multi-turn conversations where pragmatic cues and speaker intentions influence interpretation.
• Annotation: Idioms are annotated with psycholinguistic properties (familiarity, decomposability, literal plausibility) and validated by at least two native speakers.
• Evaluation Format: Each instance is paired with a multiple-choice question (MCQ) asking for the meaning, with options including the correct figurative meaning, the correct literal meaning, and two distractors.

Experimental Setup

The study evaluates a range of proprietary (GPT-5.2, Gemini 2.5 Pro) and open-source (DeepSeek-R1, Gemma-3, Llama-3.1/3.3, Mixtral, Qwen-3) models in a zero-shot setting. The evaluation decomposes performance along three axes:

1. Resource Level: High vs. Medium vs. Low.
2. Context Type: Sentence vs. Conversation.
3. Usage Type: Figurative vs. Literal.

Mechanism Analysis

To disentangle memorization from reasoning, the authors employ controlled tasks:

• Memorization Task: Models identify the figurative meaning without any context.
• Reasoning Task: Models are provided with the usage context and the English definition of the idiom, forcing them to reason about the specific usage rather than recalling a definition.
• Activation Steering: The authors apply linear activation steering (using vectors derived from MIDI and MMLU-Pro) to probe latent representations, testing if shifting models along "memorization" or "reasoning" dimensions improves performance.

3. Key Results

Performance Trends

• Resource Disparity: Performance degrades as language resources decrease. High-resource languages achieve ~59% average accuracy, while low-resource languages drop to ~50%. Open-source models show a more pronounced decline than proprietary models.
• Figurative vs. Literal: Models consistently perform better on figurative interpretations (65% average) than literal ones (48% average). This gap is most severe in high-resource languages (33 percentage points), suggesting models rely heavily on canonical figurative definitions and struggle to recognize compositional literal readings.
• Contextual Benefit: Conversational contexts improve performance over sentence-level contexts (59% vs. 54%), particularly for low-resource languages (+7 percentage points), likely due to additional pragmatic cues. However, context does not eliminate the performance gaps between resource tiers.

Memorization vs. Reasoning

• Memorization: Proprietary models exhibit high memorization capabilities across all resource levels. Open-source models show significant variation, with smaller models struggling in low-resource settings.
• Reasoning: Providing explicit definitions and context significantly boosts performance, especially for open-source models in high-resource settings. However, even with reasoning aids, gaps persist in low-resource languages, indicating that context helps compensate for limited knowledge but does not fully resolve the deficit.

Interpretation Bias

Analysis reveals a strong correlation between interpretation bias and performance. Models with a strong bias toward figurative interpretations (e.g., Gemini 2.5 Pro) achieve high figurative accuracy but suffer in literal comprehension. Conversely, models with near-neutral bias (e.g., Mixtral) perform better on literal tasks, suggesting that an over-reliance on figurative priors hinders the ability to detect literal usage.

Activation Steering

Steering activations along memorization and reasoning dimensions yields modest but consistent accuracy gains, particularly in low-resource languages. Notably, steering vectors derived from the independent MMLU-Pro benchmark transfer effectively to MIDI, suggesting a shared internal mechanism mediating the memorization-reasoning tradeoff across tasks.

Human-Model Gap

A human evaluation on a subset of the data shows native speakers achieve 94% accuracy. In contrast, the best proprietary models average 81%, and the best open-source models average 72%. The gap is most severe for open-source models in low-resource languages (e.g., Yoruba: Human 96% vs. Open-Source 23%).

4. Contributions and Significance

The paper claims the following contributions:

1. MIDI Dataset: A broad, typologically diverse testbed spanning 18 languages and dialects, featuring both sentence and dialogue contexts with literal and figurative usages.
2. Systematic Benchmarking: A comprehensive evaluation revealing that idiom comprehension remains a challenge for state-of-the-art LLMs, with specific deficits in low-resource languages and literal interpretations.
3. Mechanistic Insight: Evidence that idiom understanding is a hybrid process where memorization supplies candidate senses (priors) and reasoning leverages context for disambiguation. The study demonstrates that these capacities are intertwined; strong figurative bias can impair literal comprehension.
4. Steering Effectiveness: The demonstration that activation steering can improve performance, particularly for low-resource languages, and that this mechanism generalizes across datasets (MIDI and MMLU-Pro).

5. Limitations

The authors acknowledge several limitations:

• Scope of Analysis: The separation of memorization and reasoning focuses primarily on figurative interpretations; the interaction for literal readings (where models must suppress idiomatic meanings) is not explicitly assessed.
• Dataset Imbalance: The dataset is imbalanced across resource tiers and languages, and some languages (e.g., Vietnamese) lack literal counterparts, limiting tightly controlled cross-language comparisons.
• Annotation Noise: While reviewed by native speakers, inter-annotator agreement for psycholinguistic ratings is not reported, meaning these ratings should be treated as informative rather than definitive gold standards.
• Dialogue Naturalness: Dialogues were generated by an LLM and manually revised; while coherent, they may not fully capture the stylistic variability of natural interactions.
• Evaluation Constraints: The use of MCQs limits the assessment of open-ended interpretation and generation capabilities.

In conclusion, the paper establishes that while mLLMs have made progress, significant gaps remain in their ability to understand idioms across diverse linguistic contexts, particularly for low-resource languages and literal usages. The findings suggest that current models rely heavily on memorized priors and struggle to flexibly disambiguate meaning based on context alone.