M-QUEST -- Meme Question-Understanding Evaluation on Semantics and Toxicity

This paper introduces M-QUEST, a semantic framework and benchmark comprising 609 question-answer pairs across ten interpretive dimensions, designed to evaluate and advance the ability of large language models to perform commonsense reasoning and toxicity detection in internet memes.

The Gist

Imagine the internet is a giant, chaotic digital town square. In this square, people don't just talk; they throw around memes. A meme is like a digital joke package: it's a picture with some text on it. Sometimes, these jokes are harmless fun, like a cat wearing a hat. But sometimes, they are "toxic"—they are mean, hateful, or designed to hurt specific groups of people.

The problem is that spotting a toxic meme is incredibly hard for computers. Why? Because a meme isn't just a picture and words; it's a cultural inside joke. To understand if a meme is mean, a computer needs to know:

• Who is in the picture?
• What is the historical context?
• Is the text being sarcastic?
• Who is the joke actually at?

This paper, titled M-QUEST, is like a new "driving test" for Artificial Intelligence (AI) to see if it can understand these complex, sometimes nasty, jokes.

The Problem: The AI is Like a Literal Robot

Imagine you show a robot a picture of a clown holding a sign that says "I love kids."

• A human might look at the clown's scary makeup and realize the joke is actually about how clowns are creepy, or maybe it's a dark joke about a specific event.
• An older AI might just say, "The text says 'I love kids,' so this is a happy, safe image." It misses the vibe, the tone, and the hidden meaning.

The authors of this paper realized that current AIs are great at reading the text and seeing the objects, but they are terrible at understanding the soul of the meme. They need a better way to teach the AI how to think about these images.

The Solution: The "Meme Anatomy" Framework

To fix this, the researchers built a 10-part checklist (a framework) to dissect a meme. Think of it like a doctor performing an autopsy on a joke to see what makes it tick. The checklist includes:

1. The Basics: What does the text say? What do you see in the picture?
2. The Scene: How are things arranged? (e.g., Is someone standing over someone else?)
3. The Background Knowledge: Do you need to know about a specific celebrity, a political event, or a movie to get the joke?
4. The Emotion: Is the person in the picture angry? Sad? Is the joke trying to make us laugh or feel uncomfortable?
5. The Target: Who is the joke aimed at? Is it punching up (at the powerful) or punching down (at the vulnerable)?
6. The "Projection": Who is the viewer supposed to be? Are we supposed to feel like the victim or the bully?
7. The Analogy: Is the picture comparing two totally different things? (e.g., "This politician is like a sinking ship.")
8. The Intent: Is the creator trying to spread hate, or just be silly?
9. The Toxicity: Finally, is it actually harmful?

The Test: M-QUEST

The researchers didn't just build the checklist; they built a test called M-QUEST.

• They took 307 memes (mostly from a known dataset of "hateful memes").
• They used AI to generate 609 multiple-choice questions about these memes based on their 10-point checklist.
• Then, humans stepped in to grade the AI's questions. They asked: "Is this a good question? Is the answer actually correct?"

It's like a teacher grading a student's homework before giving it to the class. Only the best, most accurate questions made it into the final test.

The Experiment: Putting 8 AIs Through the Ringer

The researchers took 8 different "smart" AI models (like Qwen, LLaVA, and others) and asked them to take the M-QUEST test. They wanted to see: Can these AIs explain why a meme is toxic, or do they just guess?

The Results:

• The "Dumb" Robots: Some older models got less than 15% of the answers right. They were like students who didn't even read the question.
• The "Smart" Robots: The best models (specifically the newer "Qwen" family) got over 86% right.
• The Secret Sauce: The winners weren't just bigger; they were better trained to reason. They had been taught to follow instructions carefully and to think step-by-step (like a detective solving a mystery) rather than just guessing.

The Catch:
Even the smartest AI struggled with the hardest part: Pragmatic Inference.
This is the ability to understand what is implied but not said.

• Example: If a meme shows a dog looking sad next to a sign saying "Free Hugs," a human knows the dog is being sarcastic or the situation is tragic. The AI often tries to force a connection where there isn't one, or it takes the text too literally and misses the irony.

The Big Takeaway

This paper tells us that understanding internet culture is hard for computers.

1. Size isn't everything: A bigger AI isn't automatically smarter at understanding jokes. It needs specific training on how to think (reasoning) and how to listen (instruction tuning).
2. Context is King: To spot a toxic meme, you can't just look at the pixels. You need to understand the history, the culture, and the hidden meanings.
3. We need better tests: Before we can trust AI to moderate social media (to stop hate speech), we need to make sure they can pass tests like M-QUEST. Currently, they are still learning the ropes.

In a nutshell: The authors built a sophisticated "Meme School" to teach AI how to understand the difference between a funny joke and a hateful attack. They found that while the smartest students are doing well, they still need more practice understanding the subtle, unspoken rules of human culture.

Technical Summary

Here is a detailed technical summary of the paper "M-QUEST - Meme Question-Understanding Evaluation on Semantics and Toxicity."

1. Problem Statement

Internet memes are a dominant form of online communication, but their multimodal nature (combining text, images, audio, and video) and heavy reliance on commonsense knowledge make automated toxicity detection extremely difficult. Current approaches often focus on isolated elements like OCR (text) or object recognition (visuals) or use proxy variables like hate speech detection. However, there is a significant gap in:

• Holistic Understanding: A lack of a formal architecture to identify all semantic dimensions contributing to a meme's meaning.
• Explainability: Difficulty in answering why a meme is toxic, rather than just if it is toxic.
• Evaluation: Existing benchmarks often lack the granularity to connect toxicity to specific underlying semantic features (e.g., analogical mapping, background knowledge, semiotic projection).

The authors classify meme understanding as an "AI-complete problem" requiring human-like reasoning capabilities beyond standard Artificial Narrow Intelligence (ANI).

2. Methodology

The paper proposes a comprehensive framework and a corresponding benchmark, M-QUEST, developed through a semi-automatic pipeline.

A. Semantic Framework (10 Dimensions)

The authors formalized a semantic framework comprising 10 dimensions required to interpret a meme, ordered from elementary to complex:

1. Textual Material: Captions, overlays, speech bubbles, diegetic text.
2. Visual Material: People, objects, characters, scenes, graphics.
3. Scene: Configuration and spatial relationships of visual entities.
4. Background Knowledge: External context (historical events, stereotypes, cultural references) not visible in the image.
5. Overall Intent: The teleological purpose (e.g., misinformation, satire, attack).
6. Emotion: Neuro-physiological expressors (facial expressions, gestures) and affective cues.
7. Target Community: The intended audience or the group being attacked.
8. Semiotic Projection: A novel dimension identifying which element the user/observer is projected onto within the meme.
9. Analogical Mapping: The structural mapping between entities (usually visual to textual), often following an "X is Y" pattern.
10. Toxicity Assessment: The final evaluation of harm, categorized into Abusive, Cyberbullying, Harmful, Hateful, Misogynistic, and Propaganda.

B. Benchmark Construction (M-QUEST)

The benchmark was created using a neuro-symbolic framework involving two main phases:

1. Automatic Knowledge Extraction (KE): A Vision-Language Model (VLM), specifically Qwen3-VL-8b-Instruct, processes memes to extract knowledge for each of the 10 dimensions. The extraction is sequential; knowledge from earlier dimensions (e.g., Text/Visual) is fed into prompts for later, more complex dimensions (e.g., Analogical Mapping) to ensure consistency.
2. Q&A Generation: The extracted knowledge base is used to generate multiple-choice questions (one correct answer, three distractors) for each dimension.
3. Human Validation: 14 human annotators validated the generated questions and answers.
   • ToxM-QUEST: A subset of 307 memes where both the toxicity assessment and the reasoning question were validated.
   • M-QUEST (Final): The full benchmark consisting of 307 memes and 609 question-answer pairs (approx. 2 questions per meme: one on toxicity, one on a reasoning dimension).

3. Key Contributions

1. Semantic Framework: A formalized, operationalized map of 10 dimensions that encapsulates the cognitive process of meme sense-making, including the novel concept of "Semiotic Projection."
2. M-QUEST Benchmark: A compositional benchmark linking toxicity directly to underlying semantic dimensions. Unlike previous datasets that focus on binary classification, M-QUEST requires models to explain toxicity through specific reasoning dimensions.
3. Empirical Evaluation: A rigorous evaluation of 8 diverse open-weight VLMs, analyzing their performance not just on overall accuracy but on specific semantic dimensions and the consistency between toxicity judgments and reasoning.

4. Experimental Results

The authors evaluated 8 state-of-the-art VLMs (including BLIP2, InstructBLIP, LLaVA, Pixtral, and the Qwen family) in a zero-shot setting.

Performance Trends

• Overall Accuracy: Varied significantly from 13.23% (BLIP2, below random chance) to 86.38% (Qwen2.5-VL-7B-Instruct).
• Instruction Tuning & Reasoning: Models combining instruction tuning with explicit reasoning capabilities (Chain-of-Thought) significantly outperformed others.
  • Qwen Family: Achieved the highest scores (Macro-average up to 94.25%).
  • LLaVA/Pixtral: Showed moderate performance; LLaVA-v1.5 outperformed v1.6-Vicuna.
  • BLIP2/InstructBLIP: Struggled with nuanced questions due to limited reasoning and instruction alignment.
• Dimension-Specific Performance:
  • Models performed well on perceptual dimensions (Scene, Visual Material) where visual-text alignment is strong.
  • Models struggled significantly with latent dimensions requiring pragmatic inference, such as Toxicity Assessment and Overall Intent, especially when implicit relationships between text and image were required.
• Group Accuracy: When models had to answer both the toxicity and reasoning questions correctly for the same meme, performance dropped for most models, indicating a lack of cross-dimensional consistency.

Qualitative Findings

• Reasoning Patterns: Models like Qwen3 demonstrated sophisticated interpretation of irony and exclusionary dynamics. In contrast, earlier Qwen iterations (2/2.5) often misinterpreted irony as inclusivity due to a literal interpretation bias.
• Failure Modes: A critical failure mode identified was the forced alignment of visual and textual modalities. Models often attempted to construct spurious semantic connections between unrelated image elements and text, leading to incorrect toxicity assessments even when the text was understood correctly.

5. Significance and Implications

• Beyond Model Size: The results demonstrate that multimodal alignment and reasoning capabilities are more critical than parameter count alone for toxic meme detection.
• The Need for Reasoning: Toxic meme understanding is inherently a reasoning-intensive task. Models must integrate visual cues, text, background knowledge, and social context to detect latent intent.
• Benchmark Utility: M-QUEST provides a principled way to evaluate VLMs, revealing that while surface-level understanding is maturing, the ability to model latent intent and pragmatic inference remains a major bottleneck.
• Future Directions: The authors suggest that future work should focus on iterative/multi-agent QA generation to reduce human effort, refining the ontological relations between dimensions, and expanding the framework to include metadata, ethics, and legality.

Conclusion: M-QUEST establishes a new standard for evaluating multimodal safety, proving that current VLMs, even the most advanced ones, struggle with the complex, commonsense reasoning required to fully understand and explain the toxicity of internet memes.