🤖 AI

Learning to Think Like a Cartoon Captionist: Incongruity-Resolution Supervision for Multimodal Humor Understanding

This paper introduces IRS (Incongruity-Resolution Supervision), a framework that improves multimodal humor understanding by explicitly supervising the structured reasoning process of identifying visual mismatches and resolving them, thereby outperforming existing baselines on the New Yorker Cartoon Caption Contest and demonstrating that reasoning structure supervision is more critical than model scale alone.

Original authors: Hatice Merve Vural, Doga Kukul, Ege Erdem Ozlu, Demir Ekin Arikan, Bob Mankoff, Erkut Erdem, Aykut Erdem

Published 2026-04-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hatice Merve Vural, Doga Kukul, Ege Erdem Ozlu, Demir Ekin Arikan, Bob Mankoff, Erkut Erdem, Aykut Erdem

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to tell a joke.

If you just show the robot a million funny pictures and say, "Pick the funniest caption," the robot will eventually get good at guessing. It might learn that pictures with cats usually get funny comments, or that pictures of people falling down are often paired with "Ouch!" But it won't understand why it's funny. It's just memorizing patterns, like a parrot repeating words without knowing what they mean.

This paper, "Learning to Think Like a Cartoon Captionist," argues that to really understand humor, we need to stop treating it like a guessing game and start teaching the robot how to think.

Here is the simple breakdown of their new method, called IRS (Incongruity-Resolution Supervision), using a few creative analogies:

1. The Problem: The Robot is a "Black Box"

Right now, most AI models are like black boxes. You put a cartoon in one side, and a funny caption pops out the other. We don't know what happened inside. The robot might be guessing based on the color of the drawing or the length of the words, not the actual joke.

2. The Solution: The "Detective" Approach

The authors realized that human humor works like a detective solving a mystery. It follows three specific steps:

Spot the Glitch (Incongruity): "Wait a minute, that doesn't make sense! Why is a giant germ sitting in an airplane seat?"
Solve the Mystery (Resolution): "Oh! It's a pun! The germ is 'single-celled,' but it's taking up two seats like a human!"
Pick the Best Punchline (Preference): "That joke is funnier than the others because it connects the biology to the airline rules."

The paper teaches AI to do these three steps out loud, rather than just jumping to the answer.

3. The Three-Step Training Camp (IRS)

To teach the AI this "detective" mindset, they used a three-stage training process:

Stage 1: The Culture Class (Incongruity Modeling)
Imagine the AI is a foreign student who has never seen a New Yorker cartoon. They don't get the jokes. So, before teaching them to solve mysteries, the researchers put them in a "culture class." They read books, listen to podcasts, and study how professional cartoonists talk about humor. This helps the AI understand the rules of the game before it even looks at a picture.
Stage 2: The Detective's Notebook (Resolution Modeling)
This is the core. Instead of just showing the AI the answer, they show it a step-by-step diary written by a human expert.
- Bad training: "Here is a picture of a cow in a suit. The answer is 'Bovine Business'."
- IRS training: "Here is a picture. First, I see a cow in a suit. That's weird. Cows don't wear suits. Next, I see a businessman looking confused. The joke is about corporate culture. The caption 'Bovine Business' works because it mixes the animal with the office setting. Therefore, this is the best answer."
  The AI learns to write its own "detective notebook" (reasoning trace) before picking an answer.
Stage 3: The Critic's Scorecard (Preference Alignment)
Finally, the AI gets graded, but not just on whether it got the right answer. It gets graded on how it got there.
- Did it actually look at the picture, or did it guess? (Visual Grounding)
- Is the joke sound like something a real human would say, or does it sound robotic? (Style)
  It's like a cooking show where the judge doesn't just taste the food; they watch how you chopped the onions and whether you seasoned it correctly.

4. The Results: From "Guessing Parrot" to "Witty Thinker"

When they tested this method on different-sized AI models (from small to huge), the results were amazing:

Small models became much smarter, almost catching up to the big, expensive ones.
Big models became even better, reaching a level where they could rank jokes almost as well as a human expert.
The best part: Because the AI learned how to think about humor, it could apply those skills to new types of jokes it had never seen before. It didn't just memorize the training data; it learned the logic of humor.

The Big Takeaway

The paper proves that for complex, creative tasks like humor, size isn't everything. You can have a massive, powerful computer, but if it doesn't know how to think, it will fail.

By forcing the AI to slow down, spot the weirdness, solve the puzzle, and explain its reasoning, we turn it from a fortune teller (guessing the answer) into a comedian (understanding the joke).

In short: Don't just teach the robot what is funny. Teach it why it's funny, step by step.

final choice`. This forces the model to internalize the expert reasoning process rather than relying on surface-level pattern matching.

C. Preference Alignment (PA) via Humor-Aware Rewards

Goal: To ensure the generated reasoning is visually grounded and stylistically consistent with professional humor.
Approach: The model is fine-tuned using Reinforcement Learning (specifically GRPO) with a composite reward function.
Reward Components:
1. Accuracy ( $R_a$ ): Correctness of the final caption choice.
2. Format ( $R_f$ ): Adherence to the structured reasoning format.
3. Visual Perception ( $R_p$ ): A "judge" (LLM) evaluates if the reasoning explicitly references curated visual elements (e.g., specific objects, actions) to prevent hallucination.
4. Style ( $R_s$ ): A "judge" evaluates linguistic quality based on captionist guidelines (natural phrasing, punctuation, wordplay, metaphor, and punchline placement).
Optimization: The rewards for perception and style are conditioned on correctness to ensure they reinforce accurate selection rather than distracting the model.

3. Key Contributions

IRS Framework: Introduction of a theoretically grounded framework that explicitly models the "incongruity-resolution" process, bridging cognitive theory and multimodal learning.
Structured Supervision: Development of captionist reasoning traces that make intermediate reasoning steps explicit and learnable, moving beyond black-box prediction.
Humor-Aware Rewards: Design of specific reward signals (Visual Perception and Style) that guide reinforcement learning toward grounded, stylistically appropriate humor, rather than just task accuracy.
Scalability and Generalization: Demonstration that explicit reasoning supervision improves performance across model scales (7B, 32B, 72B) and transfers zero-shot to out-of-domain humor benchmarks.

4. Experimental Results

The authors evaluated IRS on the New Yorker Cartoon Caption Contest (NYCC) benchmarks across four tasks: Matching, Ranking, 10-vs-1000, and 30-vs-300.

Performance Gains:
- IRS models consistently outperformed their base counterparts (Qwen2.5-VL) and strong baselines (including closed models like o3 and o4-mini) across all scales.
- IRS-72B achieved 76.10% accuracy on the Ranking task, surpassing the closed-source o3 (62.85%) and approaching expert human performance (100% on simpler tasks, though lower on crowd-preference tasks due to the divergence between expert and crowd taste).
- On the difficult 30-vs-300 task (distinguishing between highly ranked captions), IRS-72B achieved 50.86%, significantly outperforming the base model (50.29%) and other open models.
Ablation Studies:
- Resolution Modeling (RM) was identified as the primary driver of improvement.
- Preference Alignment (PA) provided crucial gains in ranking tasks, confirming that visual grounding and stylistic quality are essential for distinguishing between plausible options.
- Incongruity Modeling (IM) alone was insufficient but provided significant boosts when combined with RM.
Zero-Shot Generalization:
- Models trained on NYCC showed strong transfer to external benchmarks like YesBut and DeepEval, improving accuracy by over 30 percentage points on some tasks. This suggests IRS learns generalizable reasoning patterns rather than dataset-specific heuristics.
Qualitative Analysis:
- IRS models produced reasoning traces that were more concise, visually grounded, and stylistically aligned with professional captionists compared to base models, which often produced verbose or superficial explanations.

5. Significance

Reasoning over Scale: The paper challenges the notion that scaling model size alone is sufficient for complex, subjective tasks. It demonstrates that supervising reasoning structure is a more effective strategy for tasks requiring deep cognitive integration like humor.
Interpretability: By forcing models to output structured reasoning traces, IRS makes the decision-making process transparent and interpretable, allowing for better debugging and analysis of model failures (e.g., visual perception errors vs. reasoning errors).
Multimodal Alignment: The work provides a blueprint for aligning multimodal models with human preferences in creative domains by using domain-specific reasoning traces and grounded reward signals, rather than generic RLHF.
Future Directions: The approach suggests that similar "structured reasoning supervision" could be applied to other subjective, creative, or culturally nuanced tasks where black-box prediction fails to capture the underlying logic.

In summary, the paper establishes that teaching multimodal models to "think like a cartoonist"—by explicitly modeling the steps of identifying mismatches, resolving them, and aligning with human stylistic preferences—leads to state-of-the-art performance in humor understanding, even surpassing larger, closed-source models.