MuSaG: A Multimodal German Sarcasm Dataset with Full-Modal Annotations

This paper introduces MuSaG, the first German multimodal sarcasm dataset featuring aligned text, audio, and video annotations from television shows, and demonstrates through benchmarking that current models struggle to match human reliance on audio cues, thereby highlighting a critical gap for future multimodal sarcasm detection research.

The Gist

Imagine you are at a dinner party. Someone says, "Oh, great weather we're having," while standing in a torrential downpour, looking miserable. You know immediately they are being sarcastic. You didn't just hear the words; you saw their rain-soaked clothes, heard the sigh in their voice, and saw the eye-roll. Your brain instantly combined all those clues to understand the real meaning.

Now, imagine trying to teach a robot to do the same thing. That is exactly what this paper, MuSaG, is about.

Here is the story of the paper, broken down into simple concepts:

1. The Problem: Robots Are "Text-Blind"

For a long time, computers have been great at reading text. If you type "I love this movie" into a computer, it thinks you are happy. But if a human says it with a groan and a face-palm, a human knows you hate it.

Current AI models are like people who can only read the menu but can't taste the food or see the chef's expression. They are terrible at catching sarcasm because they rely too much on the literal words and ignore the tone of voice (audio) and facial expressions (video).

2. The Solution: The "German Sarcasm Library" (MuSaG)

The researchers at the Karlsruhe Institute of Technology decided to build a special training library for AI. They call it MuSaG.

• What is it? It's a collection of 33 minutes of clips from German TV shows (like talk shows and comedy).
• Why German? Most AI training data is in English. The researchers wanted to see if this "sarcasm blindness" happens in other languages too.
• The Magic Ingredient: They didn't just grab random clips. They manually picked them, then had humans label them three different ways:
  1. Just the text (transcript).
  2. Just the audio (sound only, no video).
  3. Just the video (muted video, no sound).
  4. And finally, all three together.

Think of this like a cooking class where the teacher gives students the recipe (text), the sound of the sizzling pan (audio), and a video of the chef's knife skills (video) separately, so they can learn which clue is most important.

3. The Experiment: Humans vs. Robots

The researchers took this new library and tested nine different AI models (some open-source, some from big tech companies) against it. They asked the same question: "Is this sarcastic?"

Here is what they found, using a simple analogy:

• How Humans Do It: When humans listen to a sarcastic comment, they rely heavily on the voice (the tone, the pitch, the pause). It's like listening to a song; the melody tells you if it's sad or happy, even if the lyrics are neutral.
  • Result: Humans were best at detecting sarcasm using audio.
• How Robots Do It: The AI models were terrible at listening to the voice or watching the face. Instead, they acted like a robot reading a script. They ignored the tone and the eye-rolls and just looked at the words.
  • Result: The robots were best at detecting sarcasm using text.

The Big Gap: The robots are like a person trying to guess your mood by only reading your text messages, while you are standing right in front of them crying. They miss the most important clues.

4. The "Context" Trap

The researchers also tried to help the robots by giving them more background information (like the 15 seconds of conversation before the sarcastic comment).

You might think, "More context = smarter robot!"
Wrong.

It was like giving a detective a whole novel to read when they only needed to solve one specific clue. The extra information actually confused the robots. They got distracted by the surrounding chatter and started guessing wrong. It turns out, adding more noise made the task harder for them, not easier.

5. The Takeaway

The paper concludes with a clear message for the future of AI:

"We are building robots that are too good at reading and too bad at listening and watching."

To make AI that can truly understand human conversation (like a real friend), we need to stop teaching them just to read text. We need to teach them to listen to the tone and watch the face.

MuSaG is now being released to the public so other scientists can use it to build better, more "human-like" AI that doesn't miss the joke when someone says, "Oh, great," while it's raining.

Summary in One Sentence

The researchers built a German TV dataset to prove that while humans detect sarcasm by listening to voices and watching faces, current AI is still stuck reading the script, and we need to teach them to pay attention to the whole picture.

Technical Summary

Here is a detailed technical summary of the paper "MuSaG: A Multimodal German Sarcasm Dataset with Full-Modal Annotations."

1. Problem Statement

Sarcasm is a complex form of figurative language where the intended meaning contradicts the literal content, posing significant challenges for Natural Language Processing (NLP), sentiment analysis, and content moderation. While sarcasm detection has been studied extensively in text, the rise of Multimodal Large Language Models (MLLMs) necessitates the integration of audio (prosody, tone, pitch) and visual (facial expressions, gestures) cues.

Current limitations include:

• Language Bias: Most existing multimodal sarcasm datasets are in English, lacking resources for other major languages like German.
• Data Quality: Many existing datasets rely on automatically scraped social media data or lack fine-grained, modality-specific annotations.
• Evaluation Gaps: There is a lack of benchmarks that allow for the evaluation of models on isolated modalities (text-only, audio-only, video-only) versus full multimodal integration, making it difficult to understand which cues models actually utilize.

2. Methodology

A. Dataset Construction (MuSaG)

The authors introduced MuSaG, the first manually curated, human-annotated German multimodal sarcasm dataset.

• Source: Data was manually selected from four German TV shows known for sarcastic styles (Reschke Fernsehn, heute show, Die Carolin Kebekus Show, extra 3).
• Scale: The dataset contains 214 statements (120 sarcastic, 94 non-sarcastic) totaling 33 minutes of content.
• Modalities: Each instance includes aligned Text (transcripts), Audio, and Video.
• Processing:
  • Audio was sampled at 44.1 kHz; video was downsampled to 426×240 pixels at 15 fps to preserve key visual cues while reducing computational load.
  • Transcripts were generated via OpenAI Whisper 3 and manually post-edited by native German speakers.
• Annotation Protocol:
  • Participants: 12 annotators (11 native speakers, 1 proficient non-native).
  • Process: Annotators labeled statements as 'sarc' or 'non-sarc' based on specific modalities (text, audio, video, or combined).
  • Agreement: Each statement was annotated by three people. Final labels used majority voting.
  • Metrics: Inter-annotator agreement (Fleiss' Kappa) was 0.623 for multimodal and 0.594 for single-modality tasks.
  • Subsets: A subset of 155 samples with full annotator agreement (MuSaG-FullAgree) was created to serve as a gold standard.

B. Benchmarking Framework

The authors evaluated nine state-of-the-art models (8 open-source, 1 commercial) across various modality configurations:

• Models:
  • Text-only: Qwen3-8B, Qwen2.5-7B-Instruct, Qwen2-7B-Instruct.
  • Audio-only: Qwen2-Audio-7B-Instruct.
  • Vision-only: Qwen2.5-VL-7B-Instruct, Qwen2-VL-7B-Instruct.
  • Multimodal: Phi-4-multimodal-instruct, Qwen2.5-Omni-7B, Gemini-2.5-flash.
• Evaluation Configurations:
  • Unimodal: Text-only, Audio-only, Video-only.
  • Cross-modal: Text+Audio, Text+Video.
  • Multimodal: Audio+Video (realistic conversational setting), Text+Audio+Video.
• Context Analysis: The authors tested the effect of adding 15 seconds of preceding context to determine if broader discourse improves detection.

3. Key Contributions

1. MuSaG Dataset: Release of the first open, human-annotated German multimodal sarcasm dataset with independent annotations for text, audio, and video.
2. Comprehensive Benchmark: Evaluation of nine diverse models across all modality combinations, providing a baseline for future German multimodal research.
3. Human-Model Gap Analysis: A systematic comparison revealing that current models fail to leverage non-textual cues effectively compared to humans.
4. Contextual Insight: Discovery that adding conversational context often degrades model performance rather than improving it.

4. Key Results

A. Human vs. Model Performance

• Human Reliance: Humans rely most heavily on audio cues (F1: ~87.9 on FullAgree), followed by text, and then video. Prosody and intonation are the strongest indicators of sarcasm for humans.
• Model Reliance: Models perform best on text (Qwen3-8B achieved F1: 83.28).
• The Gap:
  • In audio-only tasks, humans outperform the best model (Gemini-2.5-flash) by nearly 21 F1 points.
  • In video-only tasks, humans outperform models by ~10 F1 points.
  • This indicates that current MLLMs struggle to integrate paralinguistic (audio) and visual cues effectively, relying too heavily on textual semantics.

B. Modality Combinations

• Text + Audio/Video: Adding text to audio or video generally improves model performance. Gemini-2.5-flash achieved the highest scores in these configurations (F1: 86.91 for Text+Audio).
• Audio + Video (No Text): Performance drops compared to text-inclusive setups but remains higher than single-modality baselines. This mimics real-world scenarios where transcripts are unavailable.
• Full Multimodal (Text+Audio+Video): Surprisingly, adding video to text+audio did not always improve performance and sometimes introduced noise. For example, Gemini-2.5-flash's performance dropped slightly when video was added to text+audio.

C. The Context Paradox

• Negative Impact of Context: Providing models with 15 seconds of preceding conversation significantly degraded performance for all models, often dropping them to chance levels.
• Reasoning: Models struggle to isolate the target utterance within the extended context, getting distracted by conflicting or misleading cues in the surrounding dialogue.
• Implication: Current models are brittle in realistic conversational settings where utterances are not isolated.

5. Significance and Future Directions

• Benchmarking Standard: MuSaG fills a critical gap in non-English multimodal resources, enabling research into language-specific sarcasm detection.
• Model Limitations: The study highlights a fundamental disconnect between how humans and AI perceive sarcasm. While humans use tone and expression as primary signals, models are still "text-first" systems that fail to synthesize multimodal data effectively.
• Real-World Deployment: The finding that context degrades performance suggests that deploying sarcasm detectors in real-time conversational agents (e.g., chatbots, moderation tools) requires new architectures capable of robust context handling and true multimodal fusion.
• Open Science: The dataset and code are publicly released to foster research into human-model alignment and multimodal understanding.

In conclusion, MuSaG demonstrates that while multimodal models are advancing, they have not yet achieved the nuanced, cue-integrating capabilities of humans, particularly regarding audio prosody and the handling of conversational context.