Technical Summary: ViMU - Benchmarking Video Metaphorical Understanding

1. Problem Definition

Current video understanding models predominantly focus on literal visual comprehension, such as recognizing objects, actions, and temporal relations. However, the true meaning of many videos often resides not in the overt content but in the subtext: implicit ideas, rhetorical devices, social signals, and culturally grounded meanings (e.g., irony, mockery, criticism, or political allusions) that require inference beyond what is explicitly shown.

Existing benchmarks fail to address this gap effectively because they either:

1. Target hidden spatial or physical relations rather than socially grounded meanings.
2. Focus narrowly on non-verbal humor.
3. Rely on multiple-choice formats where options may inadvertently reveal plausible subtext hypotheses, failing to enforce hint-free inference.

The paper posits that understanding video subtext requires integrating multimodal evidence, recovering culturally situated references, and inferring the creator's communicative intent without explicit cues.

2. Methodology: The ViMU Benchmark

The authors introduce ViMU (Video Metaphorical Understanding), the first benchmark designed to systematically evaluate the subtext understanding capabilities of frontier models.

Dataset Construction

• Scale: 588 videos with 2,352 questions.
• Source: Curated from platforms like YouTube, Bilibili, and TikTok.
• Curation Pipeline: A multi-stage process involving:
  1. Multimodal Evidence Extraction: Uniformly sampled frames and audio transcripts.
  2. LLM-based Taxonomy Annotation: Using frontier models to separate literal content from intended meaning, decomposing subtext into rhetorical mechanisms and social value signals.
  3. Hint-Free Question Generation: Questions are generated to be "hint-free," meaning they do not explicitly reveal the semantic dimension (e.g., sarcasm, bias) being tested, preventing shortcut learning.
  4. Iterative Validation: An iterative loop where LLMs and human experts refine questions to ensure difficulty, implicitness, and alignment with intended meaning.
• Diversity: The dataset covers over 10 rhetorical mechanisms (e.g., irony, deadpan, bait-and-switch) and social value signals (e.g., contempt, norm violation, political signaling). Evidence sources include frames, visible text, editing patterns, transcripts, and audio tone.

Evaluation Tasks

ViMU comprises four distinct tasks to probe complementary aspects of subtext understanding:

1. Open-ended Interpretation (OE): Models must explain the overall intended meaning of a video without predefined options.
2. Evidence Grounding (EG): A multiple-choice task where models select the specific multimodal elements (e.g., audio tone, editing pattern) that support their interpretation.
3. Rhetoric Mechanism Identification (RM): Models identify the rhetorical devices used to construct the message from a predefined taxonomy.
4. Social Value Signal Identification (SV): Models identify the underlying social stance or normative implication conveyed by the video.

3. Key Contributions

• First Systematic Benchmark: ViMU is the first benchmark specifically designed to evaluate video metaphorical understanding and subtext inference in a hint-free manner.
• Structured Taxonomy: It introduces a dual-dimensional framework separating how meaning is conveyed (rhetoric mechanisms) from what social meaning is expressed (social value signals).
• Comprehensive Evaluation Protocol: The benchmark includes both open-ended and multiple-choice tasks, with a specific focus on grounding interpretations in observable multimodal evidence.
• Rigorous Curation: The dataset is constructed to eliminate external context dependencies and ensure that questions cannot be answered via surface-level description alone.

4. Experimental Results

The authors evaluated 16 popular Multimodal Large Language Models (MLLMs), including both open-weight and closed-source models.

• Overall Performance: Frontier models struggle significantly with metaphorical understanding. Most models, including the most advanced closed-source ones, achieve below 50% overall performance across the four tasks.
• Task Disparity:
  • Models perform relatively better on Open-ended Interpretation (OE) and Evidence Grounding (EG), with top performers (e.g., GPT-5.2) reaching ~70%.
  • Performance drops sharply on Rhetoric Mechanism (RM) and Social Value (SV) identification, where top models often score around 20-30%.
• Model Behaviors:
  • Conservatism: Models tend to be conservative in evidence selection, often under-selecting required cues rather than hallucinating extra ones.
  • Bias: There is a systematic tendency to over-predict generic or "safer" categories (e.g., "Literal/Direct" or "Emotional Attitude") while under-predicting implicit or socially coded categories (e.g., "Implicit/Coded Social Framing" or "Identity/Ideological Signaling").
  • Family Clustering: Models from the same family exhibit similar error profiles, suggesting shared inductive biases in how they organize taxonomy labels.
  • No Universal Superiority: High performance on general video understanding tasks does not guarantee success on metaphorical subtext tasks. Furthermore, some open-weight models (e.g., Qwen3.5-27B) outperform certain closed-source models.

5. Significance and Claims

The paper claims that ViMU exposes a fundamental limitation in current video understanding systems: the inability to move beyond literal perception to interpret implicit, socially grounded meanings.

• Gap Identification: The benchmark demonstrates that frontier capabilities in general video interpretation do not automatically translate to precise understanding of implicit stance, rhetorical framing, or socially coded meaning.
• Diagnostic Utility: By providing fine-grained analyses of model behaviors (e.g., evidence grounding errors, taxonomy geometry distortion), ViMU offers insights into specific weaknesses, such as the failure to recover structured multi-source evidence patterns.
• Future Direction: The authors suggest that advancing toward robust, human-like video interpretation requires modeling not just what is shown, but what is meant. They position ViMU as a tool to guide the development of models capable of handling the complexity of social and cultural subtext in video content.

The paper concludes with a disclaimer that the dataset contains potentially offensive and harmful content, intended solely for evaluation purposes to expose model limitations in safety and social understanding, rather than as a training resource.