💬 NLP

On the Wings of Imagination: Conflicting Script-based Multi-role Framework for Humor Caption Generation

This paper introduces HOMER, a novel humor-theory-driven multi-role LLM framework that leverages script oppositions, retrieval-augmented hierarchical imagination, and targeted caption generation to significantly outperform existing state-of-the-art methods in creating funny multi-modal image captions.

Original authors: Wenbo Shang, Yuxi Sun, Jing Ma, Xin Huang

Published 2026-08-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Wenbo Shang, Yuxi Sun, Jing Ma, Xin Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to tell a joke. You might think, "Just give it a picture and ask it to be funny!" But here's the catch: for a computer, "funny" is a slippery, confusing concept. It's not just about describing what you see; it's about understanding the hidden rules of the world, spotting when those rules are broken, and then twisting your brain to find the surprise in that break. This is the heart of multimodal humor generation, a field where scientists try to help Artificial Intelligence (AI) understand the difference between a normal coffee cup and a coffee cup the size of a bathtub.

To do this, the researchers in this paper rely on a classic idea from humor theory called Script Opposition. Think of a "script" as a mental movie script we all know by heart, like "going to a meeting" or "drinking coffee." Humor often happens when two of these scripts crash into each other in a weird way—like a serious business meeting where everyone is drinking from giant mugs. The paper suggests that to make a computer truly funny, it can't just guess; it needs a structured plan to find these crashes, imagine wild connections, and then write a punchline that ties it all together.

The paper introduces a new system called HOMER (Humor-theory-driven Multi-role LLM collaboration framework with humor retrieval). Instead of asking a single AI to "be funny" and hoping for the best, HOMER acts like a tiny, specialized comedy writing team. It splits the job into three distinct roles, each played by a different AI agent working together:

The Conflict Detective (Conflicting-script Extractor): This agent looks at the image and the scene description to find the "glitch" in reality. If the image shows a meeting room with normal chairs but one giant, oversized coffee cup, this agent spots the clash between "normal office" and "giant cup." It identifies the core contradiction that makes the joke possible.
The Wild Imaginator (Hierarchical Imaginator): Once the conflict is found, this agent goes on a creative hunt. It takes the weird object (the giant cup) and builds a tree of associations. It might think: Cup → Coffee → Milk → Cow. But it doesn't just guess; it checks a massive database of real human jokes to see which connections actually land as funny. It prunes away the boring ideas and keeps the ones that have the most "humor potential."
The Punchline Writer (Caption Generator): Finally, this agent takes the conflict, the wild imagination tree, and the scene description to write the actual caption. It combines all the clues into a short, witty sentence that explains the absurdity.

The paper finds that this team approach works significantly better than letting a single AI try to do everything at once. When tested on thousands of real-world cartoon captions from the New Yorker (a famous magazine known for its funny drawings), HOMER consistently outperformed other top AI models. In fact, on average, HOMER's captions were about 7% more likely to win against human-written captions compared to the next best method.

The authors argue against the idea that simply making an AI "think harder" or "reason more" (like using complex step-by-step logic chains) is enough to generate humor. They show that without a specific guide on how humor works (like looking for script oppositions) and without a structured way to explore creative ideas, AI tends to produce captions that are logical but not actually funny. By breaking the process down into these specific, theory-driven steps, HOMER creates captions that feel more original and genuinely amusing, proving that sometimes, to be funny, you need a little bit of structure and a lot of imagination.

Technical Summary: HOMER – A Conflict-Script Based Multi-Role Framework for Humor Caption Generation

Problem Definition

The paper addresses the challenge of multi-modal humor generation, specifically the task of generating funny captions for images (e.g., cartoons). While Large Language Models (LLMs) have demonstrated capabilities in visual understanding and reasoning, they often struggle with the creativity, originality, and deep logical reasoning required for genuine humor. Existing approaches relying on generic prompting, self-improvement via reasoning chains (e.g., Chain-of-Thought), or task-specific tuning tend to capture surface-level humor language rather than the underlying logical structures of humor. Consequently, these models often produce captions that are semantically accurate but lack the "punchline" quality, interpretability, and creative imagination necessary to match human-level humor.

Methodology: HOMER Framework

The authors propose HOMER (Humor-Theory-Driven Multi-Role LLM Collaboration Framework augmented with Humor Retrieval). The framework is grounded in the General Theory of Verbal Humor (GTVH), which models humor through five knowledge resources: script opposition, situation, target, narrative strategy, and language. HOMER operationalizes this theory through three coordinated LLM-based roles:

1. Conflicting-Script Extractor

This module analyzes the input image to derive two critical components:

Situation Description ( $D$ ): A detailed contextual background including location, characters, expressions, and actions.
Conflicting Scripts ( $C$ ): The core of the humor mechanism. The extractor identifies pairs of contrasting or incongruous semantic frames (scripts) within the situation (e.g., "normal coffee cups" vs. "oversized cups"). This script opposition establishes the expectation-violation logic essential for humor.

2. Hierarchical Imaginator

Once the humor foundation ( $D$ and $C$ ) is established, this module identifies key entities as humor targets and expands the creative space through a two-pronged approach:

Deep-Pattern Imagination (Backbone Chains): The LLM performs multi-view free-associations (local and global views) to generate logical chains of related entities (e.g., coffee cups $\to$ milk $\to$ cow). These chains form the backbone of "imagination trees."
Broad-Pattern Imagination (Retrieval-Augmented): To prevent repetitive or limited outputs, the framework retrieves relevant jokes from a curated database of 335,570 one-liner jokes. It constructs queries based on the backbone entities and the humor context.
HOMER-Pruning: A scoring mechanism filters the retrieved entities to ensure high-quality imagination. The score $H(e, \varepsilon)$ $H (e, ε)$ combines three factors:
1. Relevance-Opposition ( $H_{rel}$ ): Balances semantic similarity with conceptual opposition (incongruity) using WordNet relations.
2. Humor-Frequency ( $H_{freq}$ ): Prioritizes entities that frequently appear in relevant jokes.
3. POS-Diversity ( $H_{div}$ ): Encourages lexical ambiguity (e.g., words with multiple parts of speech) to facilitate puns.

3. Caption Generator

The final module synthesizes the situation description, conflicting scripts, and a selected path from the imagination tree. It constructs a prompt $\Phi(C, D, P_i, \Omega)$ that incorporates the five GTVH knowledge resources (including narrative strategy and linguistic style options) to generate the final funny caption.

Key Contributions

Theory-Driven Mechanism: Unlike prior methods that rely on heuristic reasoning, HOMER explicitly models humor generation using the GTVH framework, providing an interpretable, step-by-step process from script opposition to caption generation.
Multi-Role Collaboration: The framework decomposes the complex task into specialized roles (Extraction, Imagination, Generation), allowing for targeted control over humor logic and materials.
Retrieval-Augmented Imagination: The introduction of a "hierarchical imaginator" that combines LLM-driven deep associations with humor-relevant retrieval from a large joke database significantly expands the creative space beyond the model's intrinsic knowledge.
Quantitative Scoring for Humor: The proposal of the $H(e, \varepsilon)$ scoring function provides a principled method to filter and select imaginative associations based on relevance, opposition, frequency, and diversity.

Experimental Results

The framework was evaluated on two New Yorker Cartoon benchmark datasets: Humor in AI and Electronic Sheep.

Performance: HOMER significantly outperformed state-of-the-art baselines (including HumorousAI, CLoT, LoL, and Phunny) and advanced reasoning strategies (CoT, Few-shot, Self-consistency).
- On the Humor in AI dataset (Top 10 group), HOMER achieved a pass@1 of 66.41%, a ~6.9% improvement over the best baseline (HumorousAI at 62.11%).
- On the Electronic Sheep dataset (High-Humor group), HOMER achieved a pass@1 of 75.53%, a ~12.1% improvement over the best baseline.
Human Evaluation: In a study with 12 raters scoring the methods, HOMER received the highest mean humor rating (3.54/5.0 on Humor in AI and 3.31/5.0 on Electronic Sheep), surpassing all baselines.
Ablation Studies: Removing any of the three core modules (Situation Description, Conflicting Scripts, or Imagination Trees) resulted in performance degradation, confirming the necessity of the full framework. The "Relevance-Opposition" score was identified as particularly critical.
Generalization: The method demonstrated robust generalization on a separate Meme dataset (ImgFlip), outperforming competitors by ~5.4% on average.
Safety: Toxicity detection using Detoxify showed negligible harmful content (sum of toxicity scores < 0.03), indicating the framework generates safe captions.

Significance and Claims

The paper claims that HOMER offers a promising path for theory-grounded multi-modal humor generation. Its primary significance lies in shifting the paradigm from "black-box" reasoning to an interpretable, modular framework that explicitly leverages humor theory. By breaking down humor generation into script extraction, imaginative expansion, and structured synthesis, the authors demonstrate that it is possible to achieve higher creativity and originality than current LLMs can produce through prompting alone. The results suggest that incorporating external humor knowledge (via retrieval) and structured theoretical guidance (via GTVH) is essential for overcoming the limitations of inherent LLM humor mechanisms. The authors remain modest, noting that the framework still struggles with purely formal or non-humorous images where script opposition is difficult to detect, acknowledging that humor remains a challenging domain for AI.