Technical Summary: HAVEN – Hierarchically Aligned Multimodal Benchmark for Unified Video Understanding

Problem Statement

While Multimodal Large Language Models (MLLMs) have demonstrated strong performance on standard video tasks such as summarization, captioning, and question answering, their ability to faithfully summarize and reason over complex narratives remains poorly evaluated. Current video benchmarks suffer from two systemic limitations:

1. Fragmented Supervision: Most existing datasets operate at a single temporal granularity (e.g., keyframes, clips, or video-level summaries), ignoring the inherently hierarchical structure of real-world videos where frames build shots, and shots construct coherent narratives.
2. Lack of Precise Cross-Modal Grounding: Text annotations are typically weakly aligned with visual content. This allows models to succeed by exploiting language priors without accurately localizing or reasoning over relevant visual evidence.

Consequently, current evaluation protocols systematically overestimate MLLM capabilities, creating an "illusion of capability" where models produce fluent text but fail to ground their outputs in correct visual evidence.

Methodology: The HAVEN Benchmark

To address these gaps, the authors introduce HAVEN (Hierarchically Aligned multimodal benchmark for unified Video undErstandiNg), a dataset and evaluation framework built on two core principles: Hierarchical completeness and Cross-modal continuity.

Dataset Construction

HAVEN is constructed by extending six existing video summarization datasets (SumMe, TVSum, OVP, YouTube, VideoXum, and MR.HiSum). The construction process involves:

• Hierarchical Annotation: Each video instance is annotated at three levels:
  • Frame Level: Uniformly sampled frames (avg. 164.9 per video) paired with LLM-generated textual descriptions ($D_V$) and human-annotated keyframes ($V_{key}$).
  • Shot Level: Frames are partitioned into temporally coherent shots ($H$) using Kernel Temporal Segmentation (KTS). Each shot has an LLM-generated description ($D_H$) and human-annotated key shots ($H_{key}$).
  • Video Level: A global document ($T$) and a concise summary ($S$) are generated by aggregating shot-level descriptions, with explicit sentence-to-shot alignments.
• Alignment: The dataset features explicit, continuous alignment between visual content and textual descriptions across all levels, ensuring language is grounded in temporally localized evidence.
• Quality Control: Human evaluation on 15 sampled videos confirmed high scores for accuracy, informativeness, and completeness across all annotation levels.

Evaluation Suite

HAVEN defines a unified suite of four task families to evaluate different aspects of video understanding:

1. Multimodal Summarization: Includes Video-to-Text (V2T), Keyframe Selection (V2V), and Joint Video-Text (V2VT) summarization.
2. Temporal Understanding: Requires models to recover the chronological order of shuffled video shots, with and without document-level context.
3. Multimodal Grounding: Tasks models to align shuffled summary sentences with their corresponding visual shots.
4. Saliency Ranking: Requires models to rank video shots by importance, with and without document context.

Key Contributions

1. Novel Dataset: HAVEN pioneers a fully granular (frame, shot, video) and fully multimodal framework with explicit, continuous cross-modal alignment, derived from existing high-quality human-annotated datasets.
2. Rigorous Testing Framework: The authors establish a multi-faceted assessment covering four core dimensions (summarization, grounding, temporal reasoning, and saliency), enabling controlled analysis of model behavior across abstraction levels.
3. Empirical Analysis: The paper provides an extensive analysis of state-of-the-art MLLMs, exposing a severe discrepancy between surface-level textual fluency and true multimodal grounding.

Experimental Results

The authors benchmarked several commercial and open-source MLLMs (including GPT-5.2, Qwen3-VL, VideoLLaMA3, and InternVideo2.5) in a zero-shot setting. Key findings include:

• The "Capability Illusion": Models consistently produce fluent textual summaries but frequently fail to ground these outputs in the correct visual evidence. High scores in text generation do not correlate with accurate visual understanding.
• Task Sensitivity: Model performance is highly sensitive to task formulation. For instance, models strong in V2T summarization often drop significantly in keyframe selection (V2V) and multimodal grounding tasks.
• Joint Generation Limitations: Joint multimodal generation (V2VT) does not consistently improve alignment between textual and visual outputs compared to independent generation. In some cases, it leads to a trade-off where text quality improves at the expense of visual selection accuracy.
• Text-Proxy Baselines: Surprisingly, text-only baselines (using frame descriptions) often outperform multimodal models in grounding and temporal reasoning tasks, suggesting that current MLLMs struggle to establish precise semantic correspondences between modalities despite having access to visual data.
• Saliency vs. Summarization: There is a clear gap between performance on summarization and saliency ranking. Strong summarization performance does not imply accurate saliency understanding, indicating that summarization relies on a more holistic notion of importance, whereas saliency ranking requires explicit visual modeling.

Significance and Claims

The paper claims that HAVEN advances the evaluation of multimodal systems beyond traditional Question Answering (QA) formats. By offering a rigorous, standardized testbed with explicit hierarchical alignment, HAVEN aims to:

• Reveal the limitations of current MLLMs that are masked by coarse or weakly grounded evaluation protocols.
• Drive future research toward interpretable, hierarchical video understanding where models can effectively ground, align, and reason over both textual and visual information in a unified manner.
• Provide a resource for developing models that move beyond "plausible" language generation to genuine multimodal comprehension.

The authors acknowledge limitations, noting that the benchmark inherits biases from source datasets, currently omits audio modalities, and focuses on relatively short videos, leaving long-form video understanding and multi-modal integration (including audio) as directions for future work.