💬 NLP

CFMS: Towards Explainable and Fine-Grained Chinese Multimodal Sarcasm Detection Benchmark

This paper introduces CFMS, the first fine-grained Chinese multimodal sarcasm dataset featuring triple-level annotations and a metaphor subset, alongside a Reinforcement Learning-augmented In-Context Learning strategy (PGDS) that significantly improves sarcasm detection and explanation generation performance.

Original authors: Junzhao Zhang, Hsiu-Yuan Huang, Chenming Tang, Yutong Yang, Yunfang Wu

Published 2026-04-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Junzhao Zhang, Hsiu-Yuan Huang, Chenming Tang, Yutong Yang, Yunfang Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to understand human humor, specifically sarcasm.

Right now, most robots are like students who only study English textbooks. They are great at spotting obvious jokes, but they get completely lost when faced with the subtle, passive-aggressive, or culturally specific sarcasm found in Chinese social media. They also tend to just guess "Yes, this is sarcasm" or "No, it isn't," without explaining why.

This paper introduces a new project called CFMS (Chinese Fine-grained Multimodal Sarcasm) to fix these problems. Here is a simple breakdown of what they did, using some everyday analogies.

1. The Problem: The Robot is "Culturally Clueless"

Think of existing sarcasm datasets as a menu at an American diner. It has burgers and fries (English jokes), but it's missing the spicy dumplings and tea eggs (Chinese sarcasm like "阴阳怪气" or passive-aggressive comments).

Furthermore, current robots are like multiple-choice test takers. They can circle "True" or "False," but if you ask them to write an essay explaining the joke, they freeze. They miss the target (who is being mocked?) and the mechanism (how is the joke built?).

2. The Solution: A New "Sarcasm School" (CFMS)

The researchers built a brand-new dataset called CFMS. Think of this as a specialized training camp for robots, designed specifically for the Chinese internet.

The Curriculum: Instead of just asking "Is this sarcastic?", they teach the robot three things:
1. Identification: "Is this a joke?"
2. Target Recognition: "Who are they making fun of? (e.g., the government, a boss, a specific person?)"
3. Explanation: "Why is it funny? Explain the conflict between the picture and the text."
The Content: They collected 2,796 real examples from Chinese social media. These aren't just random posts; they are high-quality examples where a picture and a caption clash to create a hidden meaning (like a picture of a conveyor belt with the text "Install fully consistent heads," mocking people who think exactly the same way).

3. The Secret Weapon: The "Smart Tutor" (PGDS)

Teaching a robot is hard. If you just show it random examples, it learns slowly. The researchers invented a method called PGDS (Policy-Guided Demonstration Selection).

The Analogy: Imagine you are taking a difficult math test.
- Old Way (Random 1-shot): The teacher hands you a random practice problem from last year. It might be about geometry, but your test is about algebra. It doesn't help much.
- The PGDS Way: The teacher is a smart tutor who looks at your specific question, thinks, "Ah, this looks like this specific tricky problem I solved yesterday," and shows you that exact example.
How it works: The robot uses a "reinforcement learning" strategy (basically, trial and error with a reward system) to dynamically pick the best examples to show itself before answering a new question. It doesn't need to retrain its whole brain (which is expensive); it just learns how to choose the right study guide.

4. The "Metaphor" Challenge

The researchers also tested how well these robots understand metaphors (saying one thing to mean another) versus sarcasm.

The Finding: It turns out, understanding metaphors is like climbing a mountain, while sarcasm is just hiking a hill. Even the smartest robots (like GPT-4o) struggle significantly more with metaphors. They can spot the joke, but they often miss the deeper, hidden meaning behind the imagery.

5. Why Does This Matter?

This isn't just about making robots better at jokes.

For Understanding: It helps us build AI that understands human culture and emotion deeply, not just surface-level keywords.
For Creating: They showed that if you give an AI a "sarcastic explanation" (e.g., "Make an image that mocks workplace laziness"), the AI can actually generate a funny, sarcastic image. It bridges the gap between understanding a joke and telling one.

Summary

In short, the authors built a specialized Chinese sarcasm library and a smart study method to teach AI how to get the joke, know who it's about, and explain why it's funny. They found that while AI is getting better, it still struggles with the deepest layers of human wit, especially metaphors. But with tools like CFMS and PGDS, we are taking a huge step toward AI that truly "gets" us.

1. Problem Statement

Multimodal Sarcasm Detection (MSD) has gained attention, but existing benchmarks suffer from three critical limitations that hinder the development of robust AI systems:

Cultural Bias: Most datasets are English-centric, failing to capture unique Chinese sarcastic nuances (e.g., "阴阳怪气" or passive-aggression).
Coarse-Grained Annotation: Existing benchmarks typically rely on binary classification (sarcasm vs. non-sarcasm), ignoring the specific target of the sarcasm and the mechanism of its construction.
Shallow Heuristics: Current models often rely on surface-level cues (e.g., hashtags) rather than deep, integrated multimodal reasoning required to resolve semantic conflicts between text and images.

Furthermore, there is a lack of research on how sarcasm understanding can be transferred to generative tasks (e.g., using sarcasm explanations to guide AI-generated content).

2. Methodology

A. Dataset Construction: CFMS

The authors constructed CFMS (Chinese Fine-grained Multimodal Sarcasm), the first benchmark tailored for Chinese social media.

Data Collection: A directed crawling strategy was used on Chinese social media platforms, collecting ~10,000 raw entries. A triple-stage purification pipeline (pHash deduplication, commercial filtering, and resolution filtering) reduced this to ~3,500 candidates.
Annotation Framework (Triple-Level): Unlike binary datasets, CFMS employs a three-tier annotation system:
1. Sarcasm Identification: Binary classification.
2. Target Recognition: Identifying the specific entity or phenomenon being mocked (categorized into Social Phenomena, Individual Behavior, etc.).
3. Explanation Generation: Generating natural language text explaining the semantic conflict and construction mechanism.
Human-in-the-Loop (HITL): The annotation process utilized a pipeline combining GPT-4o pre-annotation with verification by three groups of computational linguistics graduate students, achieving a Kappa coefficient of 0.69.
Parallel Subset: A high-consistency Chinese-English metaphor subset (200 entries each) was curated to study cross-linguistic metaphorical reasoning.
Total Scale: 2,796 high-quality image-text pairs.

B. Proposed Method: PGDS

To overcome the limitations of traditional retrieval-based In-Context Learning (ICL), the authors proposed Policy-Guided Demonstration Selection (PGDS).

Concept: Instead of retrieving examples based solely on static semantic similarity (cosine distance), PGDS uses a Reinforcement Learning (RL) approach to dynamically select the most effective few-shot examples (demonstrations) for a given query.
Architecture:
1. Encoding: Uses BGE for text and CLIP for images to create joint embeddings.
2. Policy Network: A lightweight two-layer MLP ( $\pi_\theta$ ) takes the query and candidate embeddings as input and outputs selection probabilities.
3. Optimization: The network is trained using the REINFORCE algorithm. The reward function ( $R$ ) is multi-dimensional, weighing format compliance, classification accuracy, target recognition accuracy, and explanation quality (via BERTScore).
Goal: To select exemplars that may have low surface similarity but high deep semantic relevance, thereby improving fine-grained understanding without modifying the base model's parameters.

3. Key Contributions

CFMS Benchmark: The first fine-grained Chinese multimodal sarcasm dataset featuring a triple-level annotation system (Identification, Target, Explanation).
PGDS Strategy: A novel, reinforcement-learned strategy for dynamic exemplar selection that outperforms static retrieval methods in complex sarcasm scenarios.
Transferability Validation: Demonstrated that fine-grained sarcasm explanations can serve as structured instruction signals for AI-Generated Content (AIGC), successfully guiding Text-to-Image models to generate content with explicit sarcastic intent (76% success rate in human evaluation).
Metaphor Analysis: Provided empirical evidence that metaphorical reasoning remains a significant bottleneck for current Multimodal Large Language Models (MLLMs), with performance dropping significantly when moving from sarcasm detection to metaphor identification.

4. Experimental Results

Performance on CFMS

Baselines: Evaluated against closed-source models (GPT-4o, Gemini-2.5-Flash) and open-source models (Qwen2.5-VL, InternVL2.5, LLaVA-1.5).
PGDS vs. Baselines:
- PGDS consistently outperformed Zero-shot, Random 1-shot, and RAG-based 1-shot baselines.
- Example: On InternVL2.5-8B, PGDS achieved 74.76% Accuracy and 50.89% Target Accuracy, surpassing RAG 1-shot (71.05% Acc).
- Efficiency: PGDS achieved performance gains comparable to or approaching LoRA fine-tuning but without the computational cost of parameter updates.
Target Recognition: Remained a bottleneck. Even fine-tuned models struggled to match the zero-shot performance of GPT-4o on target localization, indicating a need for deeper cross-modal reasoning.

Metaphor vs. Sarcasm Analysis

Difficulty Gap: Metaphor identification proved significantly harder than sarcasm detection across all models.
- In the Chinese subset, mainstream models showed an F1-score drop ( $\Delta F1$ ) of 15–20% when switching from sarcasm to metaphor tasks.
- In the English subset, GPT-4o showed an 11.47% drop.
Cultural Dependency: Open-source models with localized training (e.g., InternVL2.5) performed better on Chinese metaphors than GPT-4o, while GPT-4o maintained superiority in English contexts.

5. Significance and Future Impact

Interpretability: CFMS moves the field from simple detection to explainable understanding, providing a path for models to articulate why something is sarcastic.
Generative AI: The study bridges the gap between understanding irony and generating it, showing promise for creative applications like advertising where implicit rhetorical intent is required.
Methodological Advancement: PGDS offers a scalable solution for improving MLLM performance on complex reasoning tasks through context engineering rather than heavy fine-tuning.
Cultural Inclusivity: By focusing on Chinese social media, the work addresses the critical gap in non-English multimodal sarcasm research, highlighting unique cultural rhetorical devices.

Conclusion: The paper establishes a new standard for Chinese multimodal sarcasm detection, proving that fine-grained annotations and dynamic context selection (PGDS) are essential for advancing AI's ability to understand and generate complex, culturally-grounded semantic conflicts.