v-HUB: A Benchmark for Video Humor Understanding from Vision and Sound

This paper introduces v-HUB, a novel benchmark for video humor understanding based on non-verbal short videos with rich annotations, which reveals that current multimodal large language models struggle with visual humor alone but show improved performance when environmental audio is incorporated.

The Gist

Imagine you are trying to teach a robot how to laugh. You show it a funny video of a cat slipping on a banana peel. The robot watches, processes the pixels, hears the "thud" sound, and... says nothing. It doesn't get the joke.

This is exactly the problem v-HUB is trying to solve.

Here is a simple breakdown of the paper, using some everyday analogies.

1. The Problem: The Robot's "Funny Bone" is Broken

For a long time, AI has been great at reading jokes or looking at funny pictures. But when it comes to video, especially videos where the humor comes from actions and sounds rather than spoken words, AI struggles.

Think of current AI models like a person who has read every joke book in the library but has never actually been to a comedy club. They know the words of a joke, but they don't understand the timing, the slapstick, or the sound effects that make us laugh in real life.

2. The Solution: v-HUB (The "Laugh Lab")

The researchers created a new testing ground called v-HUB. Think of this as a specialized gym for AI, but instead of lifting weights, the AI has to "lift" its understanding of humor.

• The Workout: They collected 1,218 short, funny videos.
• The Rule: These videos are "visual-centric." This means the joke doesn't rely on someone saying a punchline. If you mute the video, it should still be funny.
• The Sources: They mixed two types of videos:
  1. Classic Silent Films: Like Charlie Chaplin movies. These are the "old masters" of visual comedy.
  2. Modern Internet Memes: Short, viral clips from today that rely on visual surprises.

3. The Three Tests (The "Gym Routine")

To see if the AI is actually getting funnier, they gave it three specific tests:

• Test A: The Caption Match (The "Punchline Quiz")
  • The Setup: The AI sees a video and five different text captions. Only one caption actually matches the joke.
  • The Challenge: The AI has to pick the creative caption that captures the humor, not just a boring description. It's like asking a robot to pick the best tweet for a funny video from a list of options.
• Test B: The Explanation (The "Why is this funny?" Test)
  • The Setup: The AI has to explain why the video is funny.
  • The Challenge: It can't just say "a cat fell." It has to say, "The cat fell because it thought the floor was solid, but it was actually a trap, and the sound of the crash made it worse." It needs to connect the dots.
• Test C: Open-Ended Questions (The "Chat" Test)
  • The Setup: The AI answers random questions about the video (e.g., "What happened before the explosion?" or "Why did the guy look shocked?").
  • The Challenge: This checks if the AI actually understands the story, not just the punchline.

4. The Big Discovery: The "Ear" Matters

Here is the most interesting part of the paper. The researchers tested the AI in three different ways:

1. Text Only: They gave the AI a written description of the video (no video, no sound).
2. Video Only: They gave the AI the video with the sound turned off.
3. Video + Audio: They gave the AI the video with the sound.

The Results:

• The "Text" AI was the smartest. When the AI could read a description, it understood the jokes best. This proves AI is still very good at language but bad at "seeing" humor.
• The "Video Only" AI was confused. Without sound, the AI missed a lot of the nuance.
• The "Video + Audio" AI was better, but still not perfect. Adding sound (like a funny "boing" sound effect or dramatic music) helped the AI understand the joke much better.

The Analogy: Imagine trying to understand a joke told in a foreign language.

• Text Only: You have a translation. You get it.
• Video Only: You see the person's face and gestures, but you can't hear the tone. You're confused.
• Video + Audio: You see the face, hear the tone, and hear the sound effects. You get most of it, but you still miss the cultural context.

5. The Conclusion: We Have a Long Way to Go

The paper concludes that current AI models are like serious students who have never been to a party. They can analyze the data, but they lack the "social intelligence" to feel the humor.

• The Good News: Adding sound (audio) helps AI understand video humor significantly.
• The Bad News: AI still relies too much on reading text descriptions. If you take away the words, the AI often stops laughing.

In a nutshell: v-HUB is a mirror that shows us AI is still a bit "tone-deaf" when it comes to visual comedy. To make AI truly funny, we need to teach it to listen to the music, watch the timing, and understand the silence, not just read the script.

Technical Summary

Here is a detailed technical summary of the paper "v-HUB: A Benchmark for Video Humor Understanding from Vision and Sound."

1. Problem Statement

While Multimodal Large Language Models (MLLMs) have advanced significantly in general video understanding, their ability to comprehend humor—a complex cognitive task requiring social reasoning, cultural context, and non-trivial inference—remains under-explored. Existing benchmarks for humor understanding suffer from several limitations:

• Linguistic Bias: Most existing datasets rely heavily on spoken dialogue or text captions, failing to test a model's ability to derive humor from visual cues alone (e.g., silent comedies).
• Neglect of Environmental Sound: Previous works often ignore non-speech audio (background music, sound effects), which is a critical component of non-verbal humor.
• Limited Scope: Existing video humor datasets often focus on short, surprising clips with low temporal complexity or are restricted to specific genres (e.g., sitcoms), lacking diversity in cultural backgrounds and humor types.
• Evaluation Gap: There is a lack of comprehensive benchmarks specifically designed to evaluate MLLMs (and OmniLLMs capable of native audio processing) on visual-centric humor reasoning.

2. Methodology

A. Dataset Construction (v-HUB)

The authors curated v-HUB, a novel benchmark consisting of 1,218 visual-centric humorous videos with an average duration of 14 seconds.

• Sources:
  • Charlie Chaplin's Silent Films (1914–1938): 729 clips ensuring humor is conveyed purely through visual actions without speech.
  • User-Generated Content: 1,8080 short videos collected from X (formerly Twitter) and YouTube, filtered to remove speech-heavy content.
• Filtering Criteria: Videos were strictly filtered to ensure:
  • Duration between 5–60 seconds.
  • No reliance on human speech (speech reliance minimized; non-English speech was muted).
  • Appropriateness (exclusion of harmful, violent, or offensive content).
• Annotation Pipeline: Eight qualified annotators performed a rigorous three-round annotation process:
  • Humor Evaluation: Rating humor levels (Low/Medium/High).
  • Captioning: Generating Descriptive Captions (factual) and Creative Captions (humor-enhancing).
  • Video Description: Detailed event descriptions focusing on observable objects/actions.
  • Humor Explanation: Iterative refinement of why the video is funny, identifying key elements (visuals, text, sound).
  • Metadata: Labeling humor sources, presence of visual text, and required background knowledge.
• Data Statistics: The dataset covers diverse humor types (Slapstick, Surprise, Misunderstanding, Irony) and cultural backgrounds (North American, East Asian, etc.). It is categorized by modality: Pure Visual (59%), Visual+Text (11%), Visual+Audio (21%), and Visual+Audio+Text (8%).

B. Evaluation Tasks

v-HUB introduces three distinct tasks to assess different facets of humor understanding:

1. Caption Matching (Discriminative): Models must select the correct "Creative Caption" for a video from a set of distractors. This tests the ability to understand the intent and creative extension of humor, not just literal description.
2. Humor Explanation (Generative): Models must identify humor points and provide coherent rationales referencing visual or auditory cues.
3. Open-ended QA: Models answer questions regarding temporal, descriptive, and causal aspects of the video to test fundamental video reasoning beyond just humor.

C. Experimental Setup

The authors evaluated a diverse set of 12 MLLMs, including:

• Specialized Video-LLMs: Qwen2.5-VL, Intern3.5-VL.
• OmniLLMs (Native Audio Processing): Qwen2.5-Omni, MiniCPM2.6-o, Video-SALMONN-2.
• Proprietary Models: GPT-4o, Gemini-2.5-Flash.

Evaluation Settings:

• Text-Only: Input is human-written video descriptions (tests language reasoning).
• Video-Only: Input is raw video frames without audio (tests visual reasoning).
• Video+Audio: Input includes video and environmental sound (tests multimodal fusion).

Metrics:

• Accuracy: For Caption Matching.
• Semantic Similarity: BERTScore, SentBERT, METEOR for generative tasks.
• AutoDQ: An entailment-based metric (Precision, Recall, F1) to verify if generated explanations cover key humor events found in ground truth.
• Human Evaluation: Qualitative rating of explanation accuracy and logicality.

3. Key Contributions

1. First Visual-Centric Video Humor Benchmark: v-HUB is the first benchmark to focus on humor derived primarily from visual cues, incorporating silent films and user-generated content, explicitly minimizing speech reliance.
2. Integration of Environmental Sound: It introduces a systematic study of how non-speech audio (music, sound effects) enhances humor understanding, a dimension largely overlooked in prior work.
3. Comprehensive Evaluation Framework: The benchmark includes discriminative (matching), generative (explanation), and reasoning (QA) tasks, providing a holistic view of MLLM capabilities.
4. Extensive Empirical Analysis: The paper provides deep insights into model failures, modality dependencies, and the impact of cultural/temporal context on humor understanding.

4. Key Results & Findings

• Heavy Reliance on Linguistic Cues: MLLMs perform significantly better in the Text-Only setting compared to Video-Only. For example, Qwen2.5-VL's SentBERT score on Open-ended QA dropped from 0.760 (Text) to 0.445 (Video). This indicates models struggle to extract humor nuances from raw visual data without textual scaffolding.
• Audio Enhances Understanding: Incorporating audio (Video+Audio) yields consistent, though modest, improvements over Video-Only. For instance, MiniCPM2.6-o improved its Caption Matching accuracy from 0.362 to 0.442 when audio was added. This confirms that environmental sound provides valuable context for humor.
• Difficulty with Subtle Humor: Models struggle with Caption Matching, particularly when the correct caption requires creative inference rather than literal description. Most models scored below 0.8 accuracy, highlighting a weakness in implicit cross-modal reasoning.
• Temporal and Cultural Bias: Models perform significantly worse on historically distant videos (Charlie Chaplin's silent films) compared to contemporary user-generated content. This suggests a gap in understanding cultural and temporal contexts of humor.
• Background Knowledge: Performance improves when explicit background knowledge is provided, suggesting that while models have implicit cultural knowledge, it is insufficient for complex humor without explicit cues.
• Visual Text Value: The presence of visual text (subtitles, on-screen text) significantly boosts performance, acting as a crucial complementary cue when audio is non-informative.

5. Significance

• Diagnosing MLLM Limitations: v-HUB exposes a critical gap in current AI: while models can describe what is happening in a video, they struggle to understand why it is funny, especially when relying solely on visual and auditory cues.
• Advancing Multimodal Reasoning: The benchmark pushes the field toward OmniLLMs that can effectively fuse visual, auditory, and textual modalities. It demonstrates that sound is not just an accessory but a necessary component for complex video understanding.
• Future Research Direction: The results highlight the need for models to develop better social intelligence, cultural awareness, and non-verbal reasoning capabilities. It sets a new standard for evaluating AI systems on tasks requiring empathy and nuanced human-like interpretation.

In conclusion, v-HUB serves as a rigorous diagnostic tool, revealing that current state-of-the-art MLLMs are still far from human-level humor comprehension, particularly in visual-centric scenarios, and underscores the necessity of integrating richer modalities and cultural context for future AI development.