Technical Summary: Computational Humor with Multimodal LLMs

Problem Statement

Multimodal humor—found in memes, cartoons, and comics—remains a significant challenge for Artificial Intelligence systems. Unlike standard multimodal tasks (e.g., image captioning or visual question answering) that rely on explicit visual and textual grounding, humor depends on non-literal mechanisms, shared cultural knowledge, and communicative intent. Current Multimodal Large Language Models (MLLMs) excel at recognizing literal content but fail to recover the metaphor, social target, or punchline encoded through incongruity between modalities. This gap represents a structural blind spot: models can identify a "burning house" or a "smiling face" but fail to interpret the satirical or ironic meaning those elements jointly encode. Existing surveys either focus solely on text-based humor or treat multimodal humor as a narrow sub-task (e.g., sarcasm detection or meme classification) without a systematic treatment of the progression from recognition to deep interpretation and generation.

Methodology and Framework

This survey organizes the literature not by model architecture or specific task names, but through a capability-centric hierarchy spanning three progressively demanding levels:

1. Recognition: The ability to detect or classify humorous phenomena (e.g., binary humor detection, target extraction, intensity scoring).
2. Interpretation and Reasoning: The ability to explain why an artifact is humorous, identifying the underlying mechanism (e.g., incongruity, analogy), the target, or the implicit meaning.
3. Generation: The ability to produce humor-consistent outputs (captions, punchlines, continuations) grounded in the input, serving as a diagnostic for whether the model has internalized humorous mechanisms.

The paper analyzes the field's evolution from task-specific fusion models to large-model approaches based on multimodal alignment, evidence-grounded reasoning, and controlled generation. It categorizes data representations into Static Visual–Textual Artifacts (StaVT) (e.g., single-image memes) and Sequential Visual Narratives (SeqVN) (e.g., multi-panel comics), noting that the latter requires tracking entities and causal relations across panels.

Key modeling paradigms identified include:

• Recognition-Oriented Alignment: Joint encoders mapping visual and textual inputs to a shared space, often via instruction tuning or contrastive pre-training.
• Interpretation-Oriented Reasoning: Methods that condition predictions on intermediate reasoning chains (rationales), utilizing techniques like Chain-of-Thought (CoT) prompting, theory-guided decomposition (e.g., incongruity-resolution), and external knowledge retrieval (RAG) to ground cultural references.
• Controllable Generation: Approaches that generate humor conditioned on control signals (mechanism, tone, target), often using cascaded pipelines (analyze-then-generate) or diffusion models for visual synthesis.

Key Contributions

The survey makes three primary contributions:

1. Capability-Centric Organization: It proposes a unified framework aligning background concepts, benchmark design, and modeling paradigms across the three capability levels (Recognition, Interpretation, Generation).
2. Synthesis of Benchmarks and Evaluation: It synthesizes existing datasets and evaluation protocols, critically analyzing what current benchmarks measure regarding interpretive understanding. It highlights that many benchmarks rely on discriminative metrics (accuracy, F1) that may not capture genuine understanding.
3. Identification of Bottlenecks: It identifies specific technical and socio-technical barriers limiting progress, including shortcut-prone evaluation, sparse mechanism-level annotation, weak cultural grounding, and unresolved safety/ownership concerns.

Empirical Results and Analysis

The authors conducted a cross-benchmark empirical analysis, evaluating a broad set of recent MLLMs (including Qwen, LLaVA, InternVL, and GPT-4o) across seven benchmarks and twelve task settings. Key findings include:

• Recognition vs. Interpretation Gap: Models generally perform well on recognition tasks (e.g., detecting humor presence or recognizing visual elements in comics), with some models achieving high accuracy on specific subtasks (e.g., GPT-4o achieving 99.21% on panel localization). However, performance drops significantly on interpretation and reasoning tasks.
• Human Performance Gap: On tasks requiring semantic or pragmatic interpretation (e.g., selecting the correct moral, title, or caption), human performance remains substantially higher than the best models. For instance, on the YesBut-v2 title selection task, humans scored 97.50% while the best model scored 83.28%.
• Reasoning Variability: "Thinking" variants of models (e.g., Qwen3-VL-8B-Thinking) showed mixed results. While they improved performance on tasks requiring social judgment (e.g., Dark Humor detection), they sometimes degraded performance on other interpretation tasks, suggesting that explicit reasoning does not automatically translate to better humor understanding.
• Model Fragmentation: No single model dominates all task types. Different models excel in different combinations of visual recognition, social knowledge, and narrative inference, indicating that humor understanding is not a monolithic capability.

Significance and Future Directions

The paper argues that the gap between perceptual recognition and communicative interpretation is a structural issue in how MLLMs are built and evaluated. The significance of this work lies in its call to move beyond "shortcut-prone" evaluation (e.g., multiple-choice questions that can be solved via surface patterns) toward rubric-based generative evaluation that assesses mechanism detection, target identification, and cultural grounding.

The authors highlight several critical challenges for future research:

• Evaluation Crisis: Moving from discriminative accuracy to open-ended, rubric-based evaluation that rewards plausible interpretations and penalizes hallucinated reasoning.
• Social and Cultural Grounding: Addressing the lack of theory-of-mind reasoning and the dominance of Western/English data, which creates blind spots for non-Western humor conventions.
• Narrative Reasoning: Developing architectures that explicitly handle temporal dependencies and entity continuity in sequential humor (comics).
• Safety and Ownership: Navigating the dual-use nature of humor generation (where the same mechanisms for wit can produce harassment) and addressing copyright/consent issues in creative datasets.

The survey concludes that achieving robust multimodal humor understanding requires a shift toward socially and rhetorically grounded reasoning, treating generation not just as a downstream application but as a diagnostic tool for interpretive competence.