Technical Summary: Easy to Complete, Hard to Choose

Problem Statement

While Large Language Models (LLMs) have demonstrated remarkable proficiency in various natural language processing tasks, significant gaps remain in understanding how these systems process culturally embedded linguistic expressions, specifically proverbs. Proverbs represent a critical testbed for language proficiency, requiring not only lexical access but also deep semantic mechanisms to navigate figurative meanings and cultural context.

The core problem investigated is whether LLMs possess genuine semantic understanding of proverbs or merely rely on statistical co-occurrence and memorized patterns. Specifically, the authors question how model performance is impacted when the task shifts from automatic pattern completion (e.g., "Better late...") to discriminative reasoning (e.g., selecting the correct ending from multiple plausible options, or identifying that none of the provided options are correct).

Methodology

Dataset: ProverbIT

The authors introduced ProverbIT, a novel Italian benchmark consisting of 100 multiple-choice questions designed to evaluate LLM reasoning on culturally grounded expressions.

• Construction: The dataset was derived from a set of 200 common Italian proverbs, with 100 selected by native Italian speakers. Each proverb was split into a beginning and a missing ending.
• Distractor Generation: For each proverb, four distinct incorrect alternative endings were manually created based on specific constructive rationales:
  • A (Assonant): Similar sounds to the original but often absurd or nonsensical.
  • B (Synonym): A literal synonym of the original ending, lacking the figurative meaning.
  • C (Inverse): The inverse of the original ending, attempting to maintain assonance.
  • D (Trivial): A tautological or trivial ending with no assonance.
• Task Design: The benchmark includes an option E ("None of the other answers"). Since all provided endings (A-D) are invented and incorrect, the correct response is always E. This forces the model to engage in discriminative reasoning rather than simple pattern matching.

Experimental Setup

The study evaluated 13 frontier models, including traditional LLMs and Large Reasoning Models (LRMs), across three distinct tasks:

1. Completion Task: Direct completion of the proverb without options (assessing knowledge retrieval).
2. Base Task (ProverbIT): Multiple-choice selection among the four incorrect options (A-D) plus option E (assessing discriminative reasoning).
3. Base + True Ending Task: Multiple-choice selection where the correct proverb ending is added as an option (assessing if performance improves when the correct answer is present).

Evaluation Metrics:

• Accuracy: Calculated as the ratio of correctly chosen options. For multiple-choice tasks, each prompt was run three times, and a majority vote determined the final answer.
• Edit Distance: Used for the completion task, with a threshold of 0.8 to determine correctness.
• Chain-of-Thought (CoT) Analysis: A detailed qualitative analysis was performed on two LRMs (DeepSeek R1 and Qwen 3) that provide full reasoning traces, examining mention frequencies of correct vs. incorrect answers and language consistency.

Key Results

Performance Disparities

The evaluation revealed a striking paradox:

• Knowledge vs. Reasoning: Nearly all models demonstrated high proficiency in the Completion Task (e.g., GPT 4o: 88%, Gemini 2.5 Pro: 94%), indicating they possess the underlying knowledge of the proverbs.
• The "Hard to Choose" Gap: Performance dropped dramatically when transitioning to the Base Task (ProverbIT) where the correct answer was absent.
  • Traditional LLMs: Suffered steep declines. For instance, GPT 4o dropped from 92% (Base + True Ending) to 64% (Base); LLama 4 Maverick plummeted from 75% (Base + True Ending) to 6% (Base).
  • Reasoning Models (LRMs): While more robust than traditional LLMs, they still exhibited substantial degradation (e.g., GPT o3 dropped from 91% to 86%; DeepSeek R1 from 89% to 74%).

Error Analysis

• Synonym Bias: Detailed error distribution showed a strong skew toward selecting Option B (literal synonyms). Models consistently preferred semantically plausible literal synonyms over recognizing the absence of the correct figurative ending.
• Model Variability: Local models (e.g., Gemma 3, Mistral Small 3.1) showed more distributed error patterns, possibly reflecting lower confidence or greater variability, whereas frontier models exhibited a specific bias toward literal interpretation.

Chain-of-Thought (CoT) Insights

Analysis of DeepSeek R1 and Qwen 3 revealed critical internal mechanisms:

• Awareness without Recognition: Both models frequently mentioned the correct proverb ending throughout their reasoning process (even though it was not an option) but failed to conclude that this correct ending was absent from the choices.
• Inconsistency: In several incorrect responses, the CoT reasoning logically pointed to the correct conclusion (Option E), yet the model output a different letter (A, B, C, or D).
• Overthinking and Language Switching: The models exhibited "overthinking" behaviors (CoTs exceeding 4,000 words) and, in the case of DeepSeek R1, inconsistent language switching between English and Italian during reasoning, potentially obscuring cultural nuances.

Key Contributions

1. ProverbIT Benchmark: The introduction of a novel Italian dataset addressing the under-representation of Italian in comprehensive LLM evaluation resources, specifically targeting culturally grounded expressions.
2. Comprehensive Evaluation: A thorough assessment of 13 frontier models (including LRMs and local models) across completion and discriminative selection tasks, highlighting the performance gap between pattern completion and reasoning.
3. CoT Analysis: A detailed investigation into the reasoning strategies of LRMs, uncovering that models often recall correct information but fail to apply it in negative reasoning tasks (identifying the absence of a correct option).

Significance and Claims

The paper claims that current LLMs rely heavily on memorized patterns rather than deeper semantic understanding of culturally grounded expressions. The significant performance degradation when the correct answer is removed suggests that models struggle with negative reasoning—the ability to determine that a correct option is not present among plausible distractors.

The authors emphasize that even state-of-the-art reasoning models exhibit limitations in grasping the underlying meaning of figurative language when forced to discriminate against plausible but incorrect alternatives. This highlights a fundamental disconnect between knowledge retrieval and discriminative reasoning capabilities. The work underscores the necessity for language-specific benchmarks that capture cultural and linguistic nuances often lost in English-centric evaluations, revealing that current systems are not yet robust in handling the complexities of figurative language comprehension.