💬 NLP

PolyFrame at MWE-2026 AdMIRe 2: When Words Are Not Enough: Multimodal Idiom Disambiguation

The PolyFrame system addresses the challenge of multimodal idiom disambiguation in the MWE-2026 AdMIRe2 shared task by employing a unified pipeline with frozen CLIP and BGE M3 encoders enhanced by lightweight modules like idiom-aware paraphrasing and sentence-type prediction, achieving significant performance gains and strong zero-shot multilingual transfer without fine-tuning large models.

Original authors: Nina Hosseini-Kivanani

Published 2026-03-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nina Hosseini-Kivanani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to understand human language, specifically the tricky parts where words don't mean what they say.

The Problem: The "Literal" Robot

Think of a standard AI model like a very smart, but extremely literal, robot chef. If you tell it, "It's raining cats and dogs," it immediately starts looking for a storm of felines and canines falling from the sky. It doesn't get the joke because it takes every word at face value.

In the real world, this is called an idiom. Idioms are phrases where the meaning is hidden (like "break a leg" meaning "good luck," not actually breaking a bone). This is hard enough in one language, but imagine trying to do this in 15 different languages at once, while also looking at pictures to see if they match the story. That was the challenge of the AdMIRe 2 competition.

The Solution: The "PolyFrame" Team

The researchers (led by Nina Hosseini-Kivanani) built a system called PolyFrame. Instead of trying to retrain the robot's entire brain (which is expensive and slow), they built a clever "translator" and "filter" system that sits in front of the robot.

Here is how their system works, step-by-step, using a simple analogy:

1. The Detective (Sentence Typing)

First, the system acts like a detective. It reads the sentence and asks: "Is this person speaking literally, or are they using a metaphor?"

If it's literal: "I saw a big fish in the pond." -> The system says, "Okay, show me a picture of a real fish."
If it's idiomatic: "He is a big fish in a small company." -> The system says, "Ah, this isn't about fish! This is about a powerful person."

2. The Translator (Idiom Rewriting)

This is the most important trick. If the detective spots an idiom, the system doesn't just guess; it rewrites the sentence for the robot.

Original: "He is a big fish."
Rewritten for the robot: "He is a very important person."
By swapping the confusing idiom for a clear, literal description, the robot can finally understand what to look for. It's like giving the robot a dictionary definition right before it tries to solve the puzzle.

3. The Double-Check (Multimodal Scoring)

The system doesn't just rely on one way of thinking. It uses three different "eyes" to look at the problem:

Eye 1 (Vision): It compares the sentence directly to the pictures.
Eye 2 (Text): It compares the sentence to the captions (descriptions) of the pictures.
Eye 3 (Hybrid): It uses a mix of both.

4. The Judge (Borda Fusion)

Finally, the system acts like a sports judge. Each "eye" gives a ranking of the pictures (1st place, 2nd place, etc.). The system then combines these rankings using a voting method called Borda Count. It doesn't just pick the winner of one eye; it looks at the overall consensus to decide which picture is the best match.

The Results: Why It Worked

The researchers started with a basic robot that got only 6.7% of the answers right on the test. It was terrible at idioms.

After adding their "Detective" and "Translator" steps:

The accuracy jumped to 60% in English.
It worked just as well in Portuguese without any extra training (Zero-Shot), proving the system was smart enough to generalize.
In the final competition with 15 languages, they achieved solid results, proving that you don't need a massive, expensive supercomputer to solve this; you just need a smart way to frame the question.

The Big Takeaway

The paper's main lesson is that you don't need to teach the robot everything from scratch. Instead, you can give it a little help by translating confusing human slang into clear, literal language first.

Think of it like this: If you want to explain a complex joke to a foreigner, you don't rewrite their entire brain's language settings. You just say, "Here is what this joke actually means," and then let them laugh. PolyFrame does exactly that for AI.

1. Problem Statement

The paper addresses the challenge of multimodal idiom disambiguation, specifically the difficulty multimodal models face in interpreting non-compositional, figurative language (idioms).

Core Issue: Standard Vision-Language Models (VLMs) like CLIP are trained on literal data and tend to interpret idioms word-by-word (literal interpretation), leading to "semantic drift" where the model selects images or captions that match the literal meaning rather than the intended figurative meaning.
Task Context: The system was developed for the AdMIRe 2.0 shared task, which involves two subtasks:
- Subtask A (Image+Text): Given a sentence containing a nominal compound and five candidate images, rank the images based on how well they reflect the intended meaning (literal vs. idiomatic).
- Subtask B (Text-Only): Given the same sentence and five candidate captions (replacing images), rank the captions.
Multilingual Challenge: The task spans 15 languages, requiring robustness across different lexicalizations and levels of transparency in idiomatic expressions.

2. Methodology: The PolyFrame System

The authors propose PolyFrame, a unified pipeline that avoids fine-tuning large multimodal encoders. Instead, it relies on frozen pre-trained models and lightweight, modular adaptations.

Architecture Overview

The system operates in four stages (see Figure 1 in the paper):

Sentence Typing: Classifies the input sentence as either "literal" or "idiomatic."
Idiom Rewrite: If the sentence is idiomatic, the target compound is replaced with a compositional paraphrase to make the figurative meaning explicit.
Similarity Scoring: Computes similarity scores using multiple zero-shot streams.
Borda Fusion: Aggregates rankings from different streams into a final output.

Key Components

Encoders (Frozen):
- Vision-Language: Uses SigLIP2 (ViT-SO400M-14-SigLIP2-378) for image-text similarity.
- Text-Only: Uses BGE M3 (multilingual dense retrieval encoder) for sentence-captions similarity.
- Note: The system does not fine-tune these heavy backbones; learning is restricted to lightweight modules.
Sentence-Type Prediction:
- Supervised: A logistic regression model trained on BGE M3 features (for English).
- Zero-Shot/Fallback: Uses "literal-first" prompting with external LLMs (e.g., GPT-4o, Qwen3-32B, Llama 3.1) to generate literal examples and classify the target sentence.
Idiom-Aware Rewriting:
- For idiomatic sentences, the system replaces the target compound with a hand-curated compositional synonym (e.g., replacing "big fish" with "important person"). This forces the VLM to align with the figurative meaning.
Distractor-Aware Scoring & Fusion:
- The system fuses three score streams using weighted Borda count aggregation:
  1. SigLIP2 Vision-Language similarity (Sentence $\leftrightarrow$ Images).
  2. BGE M3 Text similarity (Sentence $\leftrightarrow$ Captions).
  3. SigLIP2 Text similarity (Sentence $\leftrightarrow$ Captions).
- Weights: In Image+Text mode, weights are $[0.6, 0.3, 0.1]$ (favoring visual). In Text-Only mode, weights shift to $[0.0, 0.7, 0.3]$ (ignoring vision).

3. Key Contributions

Unified Pipeline: A single architecture handling both multimodal (Subtask A) and text-only (Subtask B) ranking tasks.
Lightweight Adaptation Strategy: Demonstrates that high performance can be achieved without fine-tuning massive VLMs, relying instead on idiom rewriting and sentence-type classification.
Zero-Shot Multilingual Transfer: The system achieves strong performance in Portuguese (a zero-shot language) and across 15 languages in the blind test without language-specific training data.
Ablation Insights: Identifies that idiom-aware rewriting is the primary driver of performance, while sentence-type prediction and multimodal fusion provide stability and robustness.

4. Experimental Results

The system was evaluated on the English development/test sets and a multilingual blind test (15 languages).

English Development Set (Subtask A):
- CLIP Baseline: 26.7% Top-1 accuracy (highlighting the severe literal bias).
- + Idiom Replacement: Jumped to 60.0% Top-1.
- + LLM Sentence Typing: Improved to 40.0% (when used alone), but the best configuration combined all improvements to maintain 60.0% Top-1 with higher robustness.
Zero-Shot Transfer (Portuguese):
- Achieved 60.0% Top-1 and 0.822 NDCG@5, proving the strategy generalizes well to unseen languages.
Multilingual Blind Test (15 Languages):
- Subtask A (Image+Text): Average Top-1 = 0.35, NDCG = 0.73.
- Subtask B (Text-Only): Average Top-1 = 0.32, NDCG = 0.71.
- Performance was consistent across languages, including Chinese and non-Chinese subsets.

5. Significance and Conclusion

Feasibility of Frozen Encoders: The paper challenges the notion that solving complex semantic tasks like idiom disambiguation requires fine-tuning large multimodal models. Instead, semantic framing (rewriting idioms) and modality fusion are more effective.
The "Rewriting" Effect: The most significant finding is that converting an idiomatic expression into a literal paraphrase (e.g., "big fish" $\to$ "important person") allows standard VLMs to retrieve the correct figurative image, effectively bypassing the model's literal bias.
Trade-offs: While LLM-based sentence typing improves accuracy slightly, it introduces latency and API dependencies. The authors found that a system using SigLIP2 + Idiom Replacement + Zero-Shot Classification offered the best balance of performance and efficiency for the final submission.
Limitations: The current idiom replacement database is manually curated and limited to ~50 English idioms, restricting effectiveness for rare idioms and non-English languages. Future work aims to automate this via learned fusion and encoder-decoder architectures.

In summary, PolyFrame demonstrates that effective multimodal idiom disambiguation is achievable through a modular pipeline that prioritizes semantic preprocessing (rewriting) and robust fusion over heavy model retraining.