Technical Summary: IdioLink: Retrieving Meaning Beyond Words Across Idiomatic and Literal Expressions

1. Problem Statement

Idioms and other figurative expressions pose a fundamental challenge for language models because their meaning cannot be inferred solely from their surface form or lexical composition. While recent Large Language Models (LLMs) have improved at identifying idioms and disambiguating figurative versus literal usage, it remains unclear whether these models can reliably retrieve documents based on conceptual equivalence when the surface form diverges significantly from the meaning.

Current embedding and contrastive retrieval models often rely on lexical overlap or shallow topical similarity rather than deep semantic equivalence. This leads to retrieval failures when a query uses an idiom (e.g., "break the ice") and the relevant document expresses the same concept literally (e.g., "ease the tension") or via paraphrase. Existing benchmarks do not adequately test the ability of models to align idiomatic, literal, and paraphrased expressions based on shared underlying concepts.

2. Methodology

The IdioLink Benchmark

The authors introduce IdioLink, a retrieval benchmark designed to evaluate concept-level retrieval across four distinct document types for a given Potential Idiomatic Expression (PIE):

1. Literal: The PIE is used in its compositional, non-figurative sense.
2. Idiomatic: The PIE is used in its figurative sense.
3. Simplification: A document containing the idiomatic usage but with the PIE replaced by an explicit, context-fitting explanation of its meaning.
4. Sense: A standalone document expressing the same abstract idea as the idiomatic usage without using the PIE or a direct paraphrase.

Dataset Construction:

• Scale: The dataset comprises 10,700 documents and 2,140 queries, covering 107 distinct PIEs across 10 domains (e.g., Politics, Technology, Culture).
• Annotation: Each document and query is annotated with a specific span representing the core semantic unit of the PIE.
• Validation: A subset of the test set (1,300 samples) was manually validated by three human annotators to ensure fluency, coherence, and unambiguous usage. The remainder was validated via a combination of human review and automatic validation using GPT-4o mini.
• Task Formulation: Given a query containing a marked span with a PIE (used either literally or figuratively), the model must retrieve all documents in the index whose marked spans convey the same underlying meaning.
  • If the query is literal, relevant documents are other literal instances.
  • If the query is idiomatic, relevant documents include idiomatic, simplification, and sense types.

Experimental Setup

The authors evaluated 24 embedding models (ranging from 100M to 9B parameters) under various configurations:

• Settings: Zero-shot and fine-tuned.
• Query Formulation: With and without explicit instructions guiding the model to focus on the marked span and infer the usage type.
• Embedding Granularity:
  • Sentence Embedding: Standard full-text embedding.
  • Span Embedding: A "late chunking" approach where token-level embeddings of the annotated span are aggregated via mean pooling to create a focused representation.
• Fine-tuning Strategy: A contrastive learning objective was used to align semantically equivalent expressions (e.g., literal query with literal document) while separating mismatched interpretations (hard negatives).

3. Key Contributions

1. New Retrieval Task: Introduction of a controlled retrieval task centered on idiomatic expressions to analyze conceptual similarity beyond surface form.
2. IdioLink Dataset: Release of a benchmark specifically designed to evaluate concept-level retrieval across figurative, literal, and paraphrased expressions, including a gold-standard subset.
3. Comprehensive Evaluation: An extensive analysis of modern embedding models, revealing systematic limitations in their ability to represent and retrieve shared meaning when surface forms diverge.

4. Results

Zero-Shot Performance

• Instruction Augmentation: Adding explicit instructions to queries (guiding the model to focus on the target span and usage type) substantially improved performance across all models, yielding an average increase of +11.21 in R-Precision and +14.83 in nDCG@10.
• Span Embedding: Replacing sentence embeddings with span embeddings resulted in significant gains (+12.13 R-Precision, +13.49 nDCG@10 on average), highlighting the necessity of modeling localized semantic spans for figurative language.
• Combined Approach: The combination of instructions and span embedding yielded the strongest results (+21.99 R-Precision, +22.71 nDCG@10).
• Model Scale: Parameter count alone was not a reliable predictor of performance. Smaller models (e.g., 1B parameters) often outperformed larger counterparts (e.g., 9B) when paired with effective representation strategies like span embedding.
• Baseline Comparison: While BM25 performed well on nDCG@10 due to lexical overlap with the PIE itself, it struggled with R-Precision, failing to retrieve documents where the PIE was absent (simplification and sense types).

Fine-Tuning Performance

• General Improvement: Fine-tuning consistently improved performance over zero-shot settings.
• Training Configuration: Unlike zero-shot settings, incorporating instructions during training consistently harmed performance compared to training with raw queries. Similarly, training with span embeddings underperformed compared to standard sentence embeddings.
• Inference Strategy: The most effective strategy involved training with standard methods (no instructions, sentence embeddings) but utilizing span embeddings at inference time. This suggests a decoupling of training and inference strategies is beneficial.

5. Significance and Claims

The paper claims that IdioLink exposes a critical gap in current semantic retrieval systems: the inability to consistently retrieve semantically equivalent expressions when the surface form diverges from the meaning. The authors argue that:

• Current models rely heavily on lexical overlap and topical similarity rather than genuine conceptual understanding.
• Effective idiom retrieval requires careful design of representations (specifically span-level focus) rather than simply scaling model parameters.
• IdioLink serves as a controlled testbed for studying abstraction-sensitive semantic retrieval, offering a more rigorous evaluation of figurative language understanding than previous benchmarks.

The authors maintain a modest stance, acknowledging limitations such as the focus on English, the synthetic nature of part of the corpus, and the controlled retrieval setting. They position IdioLink not as a definitive test of all idiom understanding, but as a complementary benchmark for evaluating retrieval beyond surface-level similarity.