"I See What You Did There": Can Large Vision-Language Models Understand Multimodal Puns?

1. Problem Statement

Multimodal puns are a sophisticated form of humor where visual and textual elements synergize to create meaning through polysemy (multiple meanings of a word) or phonetic similarity (homophones). While Vision-Language Models (VLMs) excel at general image captioning and understanding, their ability to comprehend the subtle cross-modal reasoning required for puns remains unexplored.

The paper identifies three critical gaps in current research:

1. Unimodal Confinement: Existing research focuses primarily on textual puns, ignoring the complex interplay where visual ambiguity contributes to the humor.
2. Benchmark Deficiencies: Current multimodal benchmarks lack rigorous negative samples (non-puns) and fine-grained categorization, making it impossible to distinguish true understanding from superficial pattern matching or "affirmative bias" (where models default to "yes" when asked if something is a pun).
3. Conflation of Preference and Comprehension: Previous evaluations often fail to separate genuine reasoning from the model's tendency to agree with the user's premise.

2. Methodology

A. Dataset Construction: MULTIPUN

The authors introduce MULTIPUN, a linguistically grounded benchmark designed to rigorously test VLMs.

• Composition: The dataset contains 445 positive pun samples and 890 adversarial negative samples (non-puns).
• Pun Types:
  • Homophonic Puns: Rely on sound similarity between words with different spellings (e.g., "pear" vs. "pair").
  • Homographic Puns: Rely on polysemy (same spelling, different meanings, e.g., "fan" as a device vs. "fan" as an enthusiast).
• Generation Pipeline:
  1. Pun Tuple Generation: Uses WordNet and the CMU Pronouncing Dictionary to identify valid word pairs ($w_p, w_a$) and their literal ($S_p$) and figurative ($S_a$) senses.
  2. Positive Sample Generation: Uses GPT-4o to generate image descriptions and captions, which are then rendered into images using GPT-image-1.
  3. Negative Sample Generation: Creates adversarial distractors using two strategies to disrupt the pun mechanism while maintaining surface coherence:
     • Explicative Substitution (ES): Replaces the pun word with a direct description of the figurative meaning, removing ambiguity.
     • Random Substitution (RS): Replaces the pun word with a semantically unrelated entity and generates a corresponding image.
• Quality Control: Includes diversity filtering (embedding-based) and human-in-the-loop verification to ensure visual-textual coherence and valid ambiguity.

B. Evaluation Tasks

The paper proposes a progressive evaluation suite to separate reasoning from bias:

1. Detection: Binary judgment (Pun vs. Non-pun).
2. Localization: Identifying the specific pun word ($w_p$) and the alternative word ($w_a$).
3. Explanation: Providing a rationale and extracting the full tuple $\langle w_p, w_a, S_p, S_a \rangle$.

• Bias Mitigation: To counter "affirmative language bias," models are queried twice: once with a "biased-to-pun" prompt and once with a "biased-to-non-pun" prompt.

C. Proposed Solutions

To address identified weaknesses, the authors propose two enhancement strategies:

1. Pun-CoT (Prompt-Level): A Chain-of-Thought prompting strategy enforcing a three-stage verification process:
   • Visual Grounding: Verify literal objects in the image to prevent visual hallucinations.
   • Lexical Anchoring: Extract exact words from the caption to prevent keyword hallucination.
   • Cross-Modal Verification: Validate phonetic or semantic bridges between text and image.
2. Pun-Tuning (Model-Level): Fine-tuning VLMs on the MULTIPUN dataset (including both puns and adversarial negatives) to improve robustness against prompt bias and enhance non-pun rejection capabilities.

3. Key Results

A. Baseline Performance

The authors evaluated 11 state-of-the-art VLMs (including GPT-5.1, Gemini-3-Pro, Claude-Sonnet-4.5, Qwen3-VL, and LLaVA).

• High False Positives: Most models struggle to distinguish puns from non-puns. They exhibit high True Positive Rates (TPR) but extremely low True Negative Rates (TNR). For example, some models classify nearly 100% of inputs as puns.
• Prompt Sensitivity: Open-source models show significant performance drops when the prompt bias shifts (high $\Delta$TPR/$\Delta$TNR), indicating their decisions are driven by prompt phrasing rather than internal reasoning.
• Explanation vs. Detection: Requiring an explanation improves TNR (models reject more non-puns) but often lowers TPR (models miss valid puns they cannot explain).
• Reasoning Models: "Thinking" variants of models show mixed results; they improve on larger models but can hallucinate more on smaller models.

B. Error Analysis

The paper categorizes model failures into four hallucination patterns:

1. Pun Word Hallucination: Ignoring the actual text to insert a word that fits a common idiom (e.g., seeing "lamp" but claiming the word is "fan").
2. Phonetic Hallucination: Forcing a sound-alike connection where none exists (e.g., claiming "banana" sounds like "soul").
3. Semantic Hallucination: Inventing meanings for words that do not exist.
4. Visual Object Hallucination: Misidentifying visual objects to align with the text (e.g., seeing an apple but claiming it is a "date" because the text says "date").

C. Impact of Proposed Solutions

• Pun-CoT: Significantly improved F1 scores across diverse architectures, primarily by boosting TNR (reducing false positives) while maintaining TPR. It effectively filters out forced associations.
• Pun-Tuning: Fine-tuning on the constructed dataset further enhanced robustness against prompt bias and improved the quality of explanations, resulting in an average 16.5% increase in F1 scores compared to vanilla models.

4. Key Contributions

1. MULTIPUN Benchmark: The first rigorous multimodal pun dataset containing 445 puns and 890 adversarial negatives, categorized into homophonic and homographic types.
2. Systematic Evaluation: A comprehensive study of 11 VLMs across detection, localization, and explanation tasks, revealing a widespread inability to distinguish genuine puns from distractors.
3. Novel Mitigation Strategies: The introduction of Pun-CoT (a structured verification prompt) and Pun-Tuning (fine-tuning with adversarial negatives), both of which significantly enhance VLM performance.
4. Error Taxonomy: A detailed classification of hallucination patterns specific to multimodal pun understanding, providing a roadmap for future model improvements.

5. Significance

This work highlights a critical limitation in current VLMs: while they can process visual and textual data, they lack the deep cross-modal reasoning required to understand human-like humor and linguistic subtleties. By providing a benchmark and effective strategies to mitigate hallucinations and bias, the paper lays the groundwork for developing more resilient VLMs capable of nuanced cultural and linguistic understanding. The findings suggest that simply scaling model size is insufficient; specific training on adversarial examples and structured reasoning (CoT) are essential for mastering multimodal ambiguity.