MetaphorVU: Towards Metaphorical Video Understanding

Technical Summary: MetaphorVU: Towards Metaphorical Video Understanding

Problem Statement
Metaphorical videos are prevalent in real-world scenarios (e.g., social media, public communication) for conveying complex ideas such as social criticism, life contemplation, and value advocacy. Understanding these videos requires high-order cognitive capabilities, specifically the ability to perform cross-domain mapping—linking perceived visual elements to underlying abstract concepts. While Multimodal Large Language Models (MLLMs) have advanced significantly in literal perception tasks (object recognition, event description), there is a lack of systematic study regarding their ability to understand metaphorical video content. This gap limits the assessment of MLLMs' high-order cognitive capabilities and hinders their reliable application in complex, real-world scenarios.

Methodology
To address this gap, the authors propose a three-pronged methodology: the construction of a benchmark, a systematic evaluation of existing models, and the development of an inference-time enhancement framework.

1. MetaphorVU-Bench Construction:

   • Taxonomy: The authors designed the first systematic video metaphor taxonomy grounded in multimodal metaphor theory and video studies. It categorizes video metaphors into eight types: Body Language, Atmosphere Language, Cultural Symbol, Naturalistic Symbol, Causal Montage, Analogical Montage, Surreal Narrative, and Performative Narrative.
   • Data Curation: The benchmark consists of 860 videos sourced from a real-world short-video platform (Kuaishou). The selection process involved a multi-stage filtration strategy:
     • Initial filtering based on audience comment volume (>150 comments).
     • LLM-based analysis of video introductions, ASR results, and comments to identify metaphorical logic.
     • MLLM-based verification to ensure visual consistency with the inferred logic.
     • Final human filtration and type identification.
   • Annotation: Human annotators provided "golden interpretations" following a fixed format (specifying visual elements and their implicit meanings) and assigned titles. Cross-validation and strict quality control (including muting videos to remove subtitle cues) ensured high-quality, reliable annotations.

2. Systematic Evaluation:

   • The benchmark was used to evaluate 11 representative MLLMs (both close-source and open-source) and various reasoning-enhanced methods (post-training and inference-time scaling).
   • Metrics: Evaluation utilized an LLM judge (DeepSeek-V3.2) to score free-form text interpretations against golden references, assessing accuracy in grounding visual elements and revealing implicit meanings.

3. MetaphorBoost Framework:

   • Motivation: Error analysis revealed that over 80% of failures were not due to recognition errors but defective cross-domain mapping (missing, superficial, or improper mapping).
   • Knowledge Graph Construction: The authors constructed the first metaphorical knowledge graph containing 54,687 nodes and 200,268 edges. This was built by extracting metaphorical concept pairs from public textual metaphor datasets and connecting related concepts.
   • Inference-Time Augmentation: MetaphorBoost operates at inference time. It first identifies visual keywords from the video, queries the metaphorical knowledge graph (using a multi-hop strategy) to retrieve relevant target concepts, and uses these retrieved concepts as external cognitive scaffolds to guide the MLLM in generating the final metaphor interpretation.

Key Results

• Performance Gap: Current MLLMs struggle significantly with metaphorical video understanding. Even the most advanced models (e.g., Gemini-3-Pro, GPT-5) achieved average scores around 64, lagging nearly 20 points behind human performance (83.4).
• Failure Analysis: The primary cause of failure is defective cross-domain mapping. Existing reasoning-enhanced methods (e.g., VideoRFT, Vision-R1) designed for recognition or event description yielded marginal or negative improvements on this task.
• Effectiveness of MetaphorBoost: The proposed MetaphorBoost framework achieved consistent performance improvements across multiple base models. For instance, it improved the average score of Gemini-3-Pro from 63.8 to 66.1 and Qwen2.5-VL-7B-Instruct from 33.8 to 37.9.
• Ablation Studies: Experiments confirmed that external knowledge is crucial, that structured knowledge graphs outperform plain text retrieval, and that metaphor-oriented knowledge is more effective than general commonsense knowledge (e.g., ConceptNet) for this specific task.

Significance and Claims
The paper claims to provide the first systematic and comprehensive benchmark dedicated to metaphorical video understanding, enabling rigorous evaluation of high-order cognitive capabilities in MLLMs. The authors assert that their work identifies cross-domain mapping as the critical bottleneck in current models and demonstrates that augmenting inference with a metaphorical knowledge graph is an effective strategy to overcome this limitation. The paper positions MetaphorVU-Bench and MetaphorBoost as foundational resources and methods for future research aimed at advancing the cognitive capabilities of MLLMs beyond literal perception.