💬 NLP

Memes-as-Replies: Can Models Select Humorous Manga Panel Responses?

This paper introduces the MaMe-Re benchmark and the Meme Reply Selection task to evaluate how well large language models can choose humorous manga panel replies, revealing that while models capture some social cues, they struggle with visual context and subtle distinctions in wit.

Original authors: Ryosuke Kohita, Seiichiro Yoshioka

Published 2026-02-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ryosuke Kohita, Seiichiro Yoshioka

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're at a lively party. Someone tells a joke about their burnt toast, and instead of just saying "Oops," you pull out a picture of a cartoon character screaming in terror. Everyone laughs. That's the power of a meme reply. It's not just about the picture; it's about how that picture fits the conversation to create a specific kind of humor.

This paper is like a detective story trying to figure out: Can computers learn to be the funniest person at the party?

Here is the breakdown of their investigation, explained simply:

1. The Big Idea: Memes are Conversations, Not Just Pictures

Most computer scientists have treated memes like static paintings in a museum. They study the painting itself: "Is this image funny? Is it offensive?"

But the authors say, "Wait a minute!" Memes are more like reaction GIFs in a text message. Their humor doesn't come from the image alone; it comes from the spark between the image and what the other person just said.

The Analogy: Think of a meme as a musical instrument. A guitar is just wood and strings (the image). It only makes music (humor) when a musician (the user) plays it in the right key (the conversation context). The computer needs to learn how to play the right song, not just identify the guitar.

2. The New Game: "Meme Reply Selection"

To test this, the researchers created a new game called Meme Reply Selection.

The Setup: They built a massive library called MAME-RE. Imagine a giant deck of cards. One side has 250 different "social media posts" (like "My socks have holes in them"). The other side has 400 different manga panels (Japanese comic strips).
The Task: For every single post, they mixed it with every single manga panel. That's 100,000 combinations!
The Human Judges: They hired over 2,000 people to look at these pairs and vote: "Is this funny?" or "Is this a disaster?" This created a "gold standard" of what humans actually find hilarious.

3. The Contenders: How Did the AI Do?

The researchers pitted two types of AI against each other to see who could pick the funniest reply.

Contender A: The "Keyword Matcher" (Similarity-based)
- How it works: This AI is like a librarian who only looks for words. If you say "socks," it looks for images with the word "socks" or "feet."
- The Result: It was okay. It rarely picked a terrible answer, but it rarely picked a brilliant one. It was safe, but boring.
- The Flaw: It missed the joke. If you said "I'm lost," it might pick a picture of a map. But the funny reply might be a picture of a confused cat. The keyword matcher didn't get the "lost" feeling; it just saw the word.
Contender B: The "Social Butterfly" (Preference-based / LLMs)
- How it works: This is a Large Language Model (like the one you are talking to now). It tries to understand the vibe, the sarcasm, and the exaggeration.
- The Result: This AI was much better! It understood that "I'm lost" could be a metaphor for life, not just geography. It picked replies that were ironic or dramatic.
- The Catch: Even the smartest AI struggled when the choices were very similar. If two pictures were both slightly funny, the AI often got confused about which one was actually the funniest.

4. The Surprising Twist: Pictures Didn't Help!

You might think, "If the AI can see the picture, it will be funnier!"

The Reality: The researchers gave the AI the actual images to look at, not just descriptions of them. It didn't help. In fact, sometimes it made things worse.
The Analogy: It's like giving a chef a recipe book (text) and then suddenly handing them a raw potato (image) without telling them how to cook it. The AI could "see" the potato, but it didn't know how to use that visual information to make a joke. It could describe the image, but it couldn't use the image to be funny.

5. The Final Verdict: The "Subtle Difference" Problem

The biggest challenge the AI faces is nuance.

The Scenario: Imagine you have three jokes. Joke A is okay. Joke B is good. Joke C is hilarious.
The Problem: To a human, Joke C is obviously the winner. To the AI, Joke A, B, and C all look very similar. The AI struggles to pick the "hilarious" one because the difference is so subtle. It's like asking a robot to pick the spiciest pepper out of three that all look exactly the same.

Summary: What Does This Mean for Us?

This paper tells us that while AI is getting better at understanding the words of a conversation, it still has a long way to go to understand the soul of a joke.

Current AI: Can pick a relevant meme (like a "thumbs up" sticker).
Future AI: Needs to learn to pick the perfect meme that makes everyone laugh out loud, understanding that humor often comes from breaking the rules, being sarcastic, or seeing the world in a weird new way.

The researchers have built the ultimate training ground (the MAME-RE benchmark) for AI to practice this skill. It's a big step toward the day when your computer assistant doesn't just answer your questions, but actually knows how to make you laugh.

1. Problem Definition

The paper addresses a gap in computational humor research: while existing studies treat memes as static artifacts (e.g., for captioning or harmfulness detection), they largely ignore the interactive function of memes as conversational replies.

The authors formalize the Meme Reply Selection task:

Input: A natural language context $c$ (e.g., a social media post) and a set of candidate memes $M = \{m_1, ..., m_{|M|}\}$ .
Goal: Select the meme $\hat{m}(c)$ that maximizes a scoring function $s(c, m)$ , representing the "funniness" of the meme in that specific context.
Core Challenge: Humor in this context is not an intrinsic property of the image but an emergent quality resulting from the juxtaposition of the meme and the conversational context (recontextualization). The task requires models to understand subtle social cues, irony, and exaggeration, rather than just semantic relevance.

2. Methodology

A. The MAME-RE Benchmark

To support this task, the authors introduced MAME-RE (Manga Meme Reply Benchmark):

Data Composition: 100,000 unique context-meme pairs derived from 250 synthetic social media posts and 400 manga panels from the openly licensed series Black Jack ni Yoroshiku.
Annotation: 2,325 unique crowdworkers provided binary annotations ("funny" vs. "not funny") for each pair, resulting in 500,000 total annotations.
Ground Truth: The funniness score for a pair is the proportion of annotators who voted it as funny.
Design Rationale: Using manga allows for a controlled study of humor via recontextualization (fixed images in new contexts) while ensuring open licensing and privacy safety (synthetic posts).

B. Evaluation Metrics

Score@1: The average reference funniness score of the selected meme.
Consensus Hit Rate (CHR): The rate at which a model selects memes that received unanimous (or near-unanimous) "funny" votes, used to filter out noisy annotations.
nDCG@5: To evaluate ranking quality.

C. Experimental Approaches

The authors compared two primary strategies:

Similarity-based Selection (sim-select): Uses embedding models (Text, LLM, Multimodal, VLM) to calculate cosine similarity between the context and meme candidates. The highest similarity is selected.
Preference-based Selection (pref-select): Uses Large Language Models (LLMs) with prompts to directly evaluate and rank candidates based on "funniness" (modeled as a multiple-choice question).

D. Experimental Design

The study proceeded through four sequential experiments:

Exp1 (Baseline): Comparing sim-select vs. pref-select across various model types.
Exp2 (Retrieve-and-Rerank): Testing a hybrid architecture where sim-select retrieves top- $k$ candidates, and pref-select chooses the best from that subset.
Exp3 (Distinguishable Candidates): A controlled test where models choose one clearly funny meme against clearly unfunny ones to establish an upper bound on capability.
Exp4 (Sensitivity to Similarity): Testing model performance when candidates are semantically similar (hard negatives) to see if models can distinguish subtle wit.

3. Key Results

Preference-based Outperforms Similarity-based: LLMs using preference-based selection (e.g., GPT-5) achieved a Score@1 of 0.325, outperforming the best similarity-based model (Sarashina-Text-Emb at 0.320). This suggests LLMs are better at capturing complex social cues like irony and exaggeration.
Visual Information is Not a Panacea: Contrary to intuition, providing visual information (either as image pixels or text descriptions of the image) did not improve performance and sometimes degraded it. This indicates a gap between understanding visual content and using it effectively for contextual humor.
The "Hard Negative" Problem:
- In Exp3, models performed well (near human levels) when the funny option was distinct from unfunny ones.
- In Exp4, performance dropped significantly as the semantic similarity between candidates increased. Models struggled to distinguish the "wittiest" reply among semantically similar options.
- Consequently, the Retrieve-and-Rerank approach (Exp2) failed to improve over the baseline. The initial retrieval step successfully gathered relevant candidates, but the reranking step failed to identify the subtle differences in humor among them.
Qualitative Analysis:
- Similarity-based models tend to be conservative, selecting semantically relevant but predictable replies.
- LLMs (Preference-based) excel at creating "personas" and using exaggeration (e.g., escalating a "3x" claim to "15x"), but they still struggle with the most subtle distinctions.

4. Key Contributions

Paradigm Shift: Proposes treating memes as interactive conversational agents rather than static content, defining the Meme Reply Selection task.
MAME-RE Benchmark: Introduces a large-scale, rigorously annotated (500k labels), open-license benchmark specifically designed for studying contextual humor in Japanese manga.
Comprehensive Analysis: Provides the first deep dive into the limitations of current multimodal models, specifically highlighting:
- The inability of current models to effectively leverage visual cues for humor generation.
- The difficulty in distinguishing subtle nuances of wit among semantically similar candidates.

5. Significance and Future Directions

Theoretical Insight: The paper challenges the assumption that adding visual modalities automatically improves multimodal reasoning. It suggests that humor relies on pragmatic alignment (recontextualization) rather than just semantic matching or visual recognition.
Practical Implication: For conversational AI to be truly socially aware, it must move beyond simple relevance matching to understand social cues, irony, and cultural context.
Future Work: The authors suggest that future research must address the "gap" between passive visual understanding and active contextual use. This may require new architectures that couple visual recognition with deep contextual inference, rather than simply scaling existing multimodal encoders. Additionally, expanding the benchmark to diverse meme formats (GIFs, photos) and cultural contexts is necessary.

In conclusion, while current LLMs show preliminary success in capturing the spirit of humorous replies, they remain significantly challenged by the subtle, high-stakes nature of selecting the funniest option among plausible candidates, particularly when visual and semantic cues are ambiguous.