MMSD3.0: A Multi-Image Benchmark for Real-World Multimodal Sarcasm Detection

This paper introduces MMSD3.0, a new multi-image benchmark for real-world multimodal sarcasm detection, and proposes the Cross-Image Reasoning Model (CIRM) with a relevance-guided fusion mechanism that achieves state-of-the-art performance across both single- and multi-image scenarios.

The Gist

Imagine you are trying to understand a joke. Usually, a joke relies on a single picture and a single caption. If the picture shows a sunny beach and the caption says, "What a terrible day for a swim," you get the sarcasm immediately. It's like a simple riddle with two pieces: Image + Text = Joke.

But in the real world, people don't just post one picture. They post a series of images to tell a story. Maybe the first photo is a pristine, expensive car, and the second photo is that same car covered in mud and broken down. The joke isn't in either picture alone; the joke is the contrast between them.

This is exactly the problem the paper "MMSD3.0" is solving.

The Problem: The "One-Picture" Blind Spot

For years, computers trying to detect sarcasm have been trained like students who only study single-page flashcards. They look at one image and the text next to it.

• The Old Way: If you showed a computer a two-part meme (like the car example above), it would get confused. It would look at the first picture and say, "Nice car!" Then it would look at the second and say, "Oh no, broken car!" It would miss the story connecting them because it wasn't built to look at the relationship between multiple images.
• The Gap: Existing datasets were like a library with only single-page books. They missed the complex, multi-page stories where the punchline happens in the transition from page 1 to page 2.

The Solution: MMSD3.0 (The New Library)

The authors built a new, massive library called MMSD3.0.

• What's in it? Instead of single pictures, every entry has 2 to 4 images (just like a real Twitter thread or an Amazon review with multiple photos).
• Where did it come from? They scraped real tweets and Amazon reviews, but they were careful to remove "cheating" clues (like hashtags that literally say #sarcasm) so the computer has to actually think to find the joke.
• The Twist: They even used AI to generate fake sarcastic posts to fill in the gaps, ensuring the dataset is huge and diverse.

The New Detective: CIRM

To read this new library, they built a new AI detective called CIRM (Cross-Image Reasoning Model). Think of CIRM as a detective who doesn't just look at clues in isolation but connects the dots.

Here is how CIRM works, using a simple analogy:

1. The "Bridge" (Dual-Stage Bridge Module):
   Imagine you are reading a comic strip. You need to understand how Panel A leads to Panel B. CIRM builds a bridge between the images. It doesn't just look at Image 1 and Image 2 separately; it asks, "How does the text in Image 1 change the meaning of Image 2?" It creates a conversation between the pictures.

2. The "Spotlight" (Relevance-Guided Fusion):
   Sometimes, a post has 4 images, but only 2 of them are actually part of the joke. The other two might be random filler. CIRM has a spotlight that shines only on the images that matter. It ignores the "noise" and focuses its brainpower on the images that actually hold the sarcastic punchline.

3. The "Order Matters" (Positional Encoding):
   In a story, the order of events is crucial. If you show the "broken car" before the "nice car," it's a tragedy. If you show the "nice car" before the "broken car," it's a sarcastic joke about bad luck. CIRM remembers the order of the images, ensuring it doesn't mix up the timeline.

Why This Matters

Before this paper, AI was like a person who could only understand jokes told in a single sentence. If you told a joke that required a sequence of events, the AI would miss it.

• The Result: The new model (CIRM) is much smarter. It can now understand complex, multi-image sarcasm, just like a human does.
• The Proof: When they tested it, the old models (trained on single images) failed miserably on the new multi-image dataset. But CIRM aced the test, proving that to understand human humor in the real world, you have to look at the whole picture, not just one frame.

In a Nutshell

The authors realized that sarcasm is often a team sport involving multiple images working together. They built a new training ground (MMSD3.0) and a new coach (CIRM) that teaches AI to look at the whole team, understand the relationships between the players, and finally get the joke.

Technical Summary

1. Problem Statement

Multimodal sarcasm detection (MSD) has traditionally focused on single-image scenarios (e.g., a tweet with one image and text). However, real-world social media content frequently contains multiple images (up to four on Twitter), where sarcasm often arises from the semantic or affective contrast between images rather than just the text-image mismatch.

• The Gap: Existing datasets (MMSD, MMSD2.0) and methods are limited to single images, failing to capture "inter-image" dependencies.
• The Challenge: Detecting sarcasm in multi-image settings requires modeling complex relationships (e.g., a narrative progression or a visual juxtaposition) and handling longer, more diverse textual contexts found in real-world posts.

2. Key Contributions

A. MMSD3.0 Dataset

The authors introduce MMSD3.0, the first benchmark specifically designed for multi-image sarcasm detection.

• Composition: Contains over 10,000 instances curated from Twitter and Amazon reviews.
• Structure: Each sample includes 2 to 4 images (the maximum allowed on Twitter) and associated text.
• Quality Control:
  • Source Diversity: Unlike previous datasets relying on specific hashtags (which introduce bias), MMSD3.0 uses unfiltered tweets and reviews to ensure fairness.
  • Enrichment: Retains emojis (often removed in prior work) and includes OCR text extracted from images, as these are crucial for sentiment cues.
  • Annotation: Annotated by 9 experts with a Cohen's Kappa of 0.816.
  • AI Augmentation: Used Qwen2.5-VL-32B to generate synthetic sarcastic candidates and GPT-4o to evaluate them, expanding the dataset with high-quality, diverse samples.
• Statistics: The dataset features longer text (avg. ~31 words vs. ~15 in MMSD) and higher OCR/emoji coverage, better reflecting real-world complexity.

B. CIRM Model (Cross-Image Reasoning Model)

To address the multi-image challenge, the authors propose CIRM, a framework designed to model inter-image dependencies and cross-modal correspondences. Its architecture consists of five key components:

1. Data Encoding:
   • Visual: Uses ViT to encode up to 4 images (padding with blanks if fewer).
   • Text/OCR: Uses RoBERTa-Emoji for text and PP-OCRv5 for extracting text from images.
2. Positional Encoding & Masking:
   • Adds positional embeddings to image features to preserve image order, which is critical for narrative or contrastive sarcasm.
   • Uses padding masks to ignore non-existent images.
3. Dual-Stage Bridge Module (DSBM):
   • Pre-Bridge: Performs cross-modal attention (Text $\leftrightarrow$ Image) before sequential modeling.
   • Sequential Modeling: Uses a Mamba-inspired state-space model (with Conv1D and selective state updates) to capture long-range dependencies within each modality independently.
   • Post-Bridge: Re-establishes cross-modal alignment after sequence modeling to refine joint semantics.
4. Relevance-Guided Fusion Module (RGFM):
   • Aligns text and visual features using OCR cues.
   • Computes a relevance score for each image based on cosine similarity and a learnable MLP, adaptively weighting images to suppress noise from irrelevant or padded images.
5. Classification: Aggregates global pooled features from text, vision, and the relevance-guided fusion vector for final prediction.

3. Experimental Results

Datasets Evaluated

• MMSD & MMSD2.0: Used to validate CIRM's performance on single-image tasks (transferability).
• MMSD3.0: Used to evaluate multi-image performance.

Performance Highlights

• Single-Image (MMSD/MMSD2.0): CIRM achieved State-of-the-Art (SOTA) results, outperforming previous bests by ~1.5% on MMSD2.0 (92.12% Acc, 91.69% F1). This proves the model is robust even in standard scenarios.
• Multi-Image (MMSD3.0):
  • CIRM: Achieved 85.16% Accuracy and 84.42% F1.
  • Baselines: Traditional single-image methods (e.g., Multi-view CLIP, MoBA) and large multimodal models (GPT-4o, LLaVA-1.5, Qwen2.5-VL) performed significantly worse (F1 scores around 71-82%).
  • Ablation: Removing the Dual-Stage Bridge or Relevance-Guided Fusion caused the largest performance drops, confirming their necessity. Removing positional encoding also degraded performance, proving the importance of image order.
• Real-World vs. AI-Generated: CIRM maintained robust performance on real-world data (80.39% F1), whereas other models struggled, highlighting CIRM's ability to handle the unpredictability of real-world sarcasm.

4. Significance and Impact

• Bridging the Gap: MMSD3.0 is the first dataset to systematically address the "multi-image gap" in sarcasm detection, moving the field closer to real-world applicability.
• Methodological Advancement: The CIRM architecture demonstrates that explicit modeling of inter-image relationships (via the Dual-Stage Bridge) and adaptive fusion (via RGFM) is superior to simple concatenation or tiling of images.
• Practical Utility: The inclusion of OCR and emojis, along with the focus on longer contexts, makes this benchmark and model more suitable for deployment in real-world social media monitoring and content moderation systems.
• Open Science: The dataset and code are publicly available, fostering further research into complex multimodal reasoning.

In conclusion, the paper establishes that sarcasm in the real world is often a multi-image phenomenon. By introducing MMSD3.0 and the CIRM framework, the authors provide both the data and the architectural tools necessary to detect these complex, cross-image ironic cues effectively.