I Came, I Saw, I Explained: Benchmarking Multimodal LLMs on Figurative Meaning in Memes

1. Problem Statement

Internet memes are a dominant form of multimodal communication that frequently rely on figurative language (e.g., irony, metaphor, sarcasm) to convey layered, non-literal meanings. While Multimodal Large Language Models (MLLMs) have advanced significantly in general vision-language tasks, their ability to detect and explain figurative meaning in memes remains poorly understood.

Key gaps identified by the authors include:

• Lack of Systematic Evaluation: Previous work focused mostly on text-centric figurative language or simple meme classification (e.g., hate speech, sentiment). There is limited research on how MLLMs handle the specific interplay of text and images required for figurative interpretation.
• Explanation Quality: Existing benchmarks often focus solely on label accuracy (classification) without evaluating the faithfulness or coherence of the model's generated explanations.
• Modality Contribution: It is unclear whether models rely more on visual or textual cues for different types of figurative meaning (e.g., does irony rely more on the image or the text?).

2. Methodology

2.1 Datasets

The study evaluates models on three publicly available datasets, focusing on memes with embedded text:

• FigMemes: A multi-label dataset of politically opinionated memes annotated for six types of figurative meaning: Allusion, Exaggeration, Irony/Sarcasm, Anthropomorphism, Metaphor/Simile, and Contrast.
• Memotion 2: Focuses on sarcasm detection (annotated as levels of sarcasm: None, Little, Very, Extremely).
• MET-Meme: Focuses on metaphor detection (binary: Literal vs. Metaphoric).

2.2 Experimental Setup

• Models: Eight state-of-the-art MLLMs from three families were evaluated in zero-shot settings:
  • Aya-Vision: 8B, 32B
  • Gemma 3: 4B, 12B, 27B
  • Qwen2.5-VL: 7B, 32B, 72B
• Task Definition: The authors define a combined Detection and Explanation task. Models must:
  1. Predict the presence/absence (or degree) of specific figurative types.
  2. Generate a natural language explanation justifying the prediction.
• Prompting Strategies:
  1. Label-based: Binary classification (Yes/No) with explanation.
  2. Score-based: Continuous probability score (0–1) with explanation.
• Modality Ablation: To isolate the contribution of each modality, the authors created three input conditions:
  • Original Meme: Full text and image.
  • Text-Only: Image content masked; only embedded text provided.
  • Image-Only: Text removed via OCR and inpainting (using PaddleOCR and LaMa); only visual content provided.

2.3 Evaluation Metrics

• Automatic Evaluation: F1 scores (Micro and Macro) for classification tasks.
• Human Evaluation: A study involving two annotators rating 90 sampled memes on a 5-point Likert scale based on seven criteria:
  • Label-Explanation Consistency: Relevance and internal consistency.
  • Content Faithfulness: Correctness, appropriate length, visual/textual completeness, and absence of hallucinations (faithfulness).

3. Key Contributions

1. New Benchmark Task: Introduced a rigorous detection and explanation framework for figurative meaning in memes, moving beyond simple classification.
2. Modality Analysis: Systematically quantified how removing text or images affects performance across different figurative types, revealing distinct modality dependencies.
3. Faithfulness Gap Analysis: Conducted a comprehensive human study revealing that high classification accuracy does not guarantee faithful or coherent explanations.
4. Bias Discovery: Identified a pervasive bias where models over-predict figurative meaning even in literal memes.

4. Key Results & Findings

4.1 Detection Performance

• Model Size: Larger models generally outperform smaller ones, with Qwen-72B achieving the best overall results.
• Type-Specific Performance:
  • Contrast was the easiest type to detect (highest F1).
  • Anthropomorphism and Allusion were the most difficult.
  • Irony/Sarcasm and Metaphor showed the largest performance gaps between models.
• Modality Dependencies:
  • Visually Grounded: Irony, Sarcasm, Allusion, Exaggeration, and Anthropomorphism rely heavily on visual cues. Performance dropped significantly when images were removed.
  • Textually Grounded: Contrast relies more on text.
  • Interplay Required: Metaphor detection uniquely requires a complex interplay of both text and image; performance dropped drastically when either modality was removed.

4.2 The "Over-Assignment" Bias

• Models exhibit a strong bias toward assigning figurative labels to memes that are actually literal.
• In the multi-label task, models frequently predicted 1–3 figurative types for memes containing zero figurative labels.
• Paradoxically, for literal memes, models sometimes performed better with ablated inputs (text-only or image-only) than with the full meme, suggesting the full "meme format" triggers a bias toward finding figurative meaning.

4.3 Explanation Quality (Human Evaluation)

• Faithfulness Gap: Even when models predicted the correct label, their explanations often failed to be faithful to the content.
• Visual Hallucinations: The primary failure mode was visual hallucination (describing elements not present) or over-interpretation (forcing a figurative meaning where none exists).
• Text vs. Image: Models were better at capturing textual elements than visual ones.
• Behavioral Reasoning: Models struggled with memes relying on socio-psychological nuances (e.g., consumer behavior irony), often missing the underlying human context.
• Subjectivity: In some cases, models assigned "wrong" labels (relative to the gold standard) but provided valid, evidence-based explanations, highlighting the inherent subjectivity of figurative interpretation.

5. Significance and Conclusion

This paper provides a critical reality check for the current capabilities of MLLMs in understanding complex, non-literal communication.

• Beyond Accuracy: The study demonstrates that high classification accuracy is insufficient; explanation quality is a crucial metric for trust and usability.
• Modality Balance: It highlights that current models do not yet possess a robust, unified understanding of how text and image interact to create meaning, particularly for metaphors.
• Future Directions: The authors call for future research to move beyond simple label prediction toward developing architectures that ensure reasoning faithfulness and better capture the socio-psychological contexts inherent in human communication.

The work underscores that while MLLMs are powerful, they currently struggle with the nuance, subjectivity, and visual-textual integration required to truly "understand" internet memes.