Technical Summary: MuseBench

Problem Statement

Current Multimodal Large Language Models (MLLMs) have demonstrated strong capabilities in perceptual recognition and factual reasoning regarding video content. However, a critical gap remains in intent-level artistic understanding. Understanding audiovisual arts (cinema, static visual arts, stage performance, and game design) requires reasoning not merely about what is depicted, but why specific creative choices (e.g., framing, lighting, editing pace, sound design) were made to convey emotion, theme, or aesthetic intent.

Existing video benchmarks largely evaluate surface-level recognition or temporal reasoning using single-choice questions with a single correct answer. This format fails to capture the plurality of valid interpretations inherent in artistic analysis, where multiple defensible perspectives can coexist. Furthermore, constructing benchmarks for this domain is hindered by the scarcity of expert-knowledge data and the difficulty of generating high-quality distractors that do not rely on transcript-based shortcuts.

Methodology

1. Data Source: Video Essays

To address the scarcity of expert-knowledge data, the authors utilize video essays (sourced from YouTube, Bilibili, and TikTok) as the primary knowledge base. These videos pair professional commentary with visual demonstrations, providing a natural alignment between expert reasoning and visual evidence. The dataset is distilled from over 10,000 candidate video essays.

2. Construction Pipeline

The authors propose a four-phase iterative construction pipeline to transform raw video essays into benchmark questions:

• Segmentation: Videos are partitioned into 10-second intervals.
• Clip Captioning: A vision-language model (Keye-VL-1.5) generates fine-grained captions for each segment, conditioned on the narrator's transcript. These captions serve as construction resources but are never exposed to the evaluated models.
• Question Generation: Questions are generated in two formats:
  • Single-Select: Probes discrete recognition of the most precise interpretation (1 correct answer among 4–8 options).
  • Multi-Select: Probes set-valued analytical judgment, acknowledging that multiple artistic intentions can be valid simultaneously (2–4 correct answers among 4–8 options).
• Distractor Generation: An adversarial step generates plausible distractors using seven strategies (e.g., technical misread, over-simplification, factual error, conceptual confusion, scope error, temporal confusion, partial truth). This ensures all options appear equally plausible to a reader without access to the clip, suppressing shortcut-based guessing.

3. Quality Review

A human-in-the-loop iterative review process refines the questions. Domain experts rate items on four dimensions (Holistic Quality, Visual Necessity, Mechanistic Trace, Answer Integrity) and tag failure modes (e.g., narrator-dependent answerability, ambiguous stems). The generation prompts are updated iteratively to eliminate these failures until the benchmark meets quality standards.

4. Evaluation Protocol

To handle variable option counts and the open-ended nature of artistic reasoning, the paper introduces specific metrics:

• Chance-Adjusted Accuracy (CAA): For single-select questions, this normalizes scores to account for varying numbers of options ($K_i$), ensuring random guessing yields a score of 0 and a correct answer yields 1, regardless of $K_i$.
• Set-Based Metrics (Precision, Recall, F1): For multi-select questions, exact-match accuracy is deemed too strict. The authors report Precision, Recall, and F1 scores on the predicted set of options to capture partial agreement and distinguish between models that identify the most salient option versus those that capture the full breadth of valid interpretations.

Key Contributions

1. MuseBench Benchmark: A comprehensive benchmark comprising 4,016 expert-validated questions across four art categories (Cinematic Arts, Static Visual Arts, Stage Performing Arts, Game Arts) and 11 sub-domains. It uniquely combines single-select and multi-select formats with variable option counts to model interpretive plurality.
2. Scalable Expert-Knowledge Pipeline: A novel four-phase pipeline that leverages video essays and LLMs to generate visually grounded, intent-level questions at scale, addressing the fundamental challenge of acquiring expert knowledge for creative domains without relying on crowdsourcing.
3. Principled Evaluation Protocol: A heterogeneous-format scoring protocol utilizing Chance-Adjusted Accuracy and set-based F1 with exact-match diagnostics. This allows for a more nuanced assessment of model capabilities in artistic reasoning compared to standard accuracy metrics.

Results

The authors conducted zero-shot evaluations of 28 state-of-the-art MLLMs (including proprietary and open-source models) on MuseBench.

• Performance Gap: Even the best-performing model (Claude-4.6-Opus) achieved only 48.29% overall accuracy, substantially below the 87.18% performance of human experts. This exposes a significant gap in current models' creative domain expertise.
• Category-Specific Weaknesses: Models consistently lagged sharply on Game Arts across all tiers, likely due to the limited representation of interactive visuals and real-time camera control in pretraining corpora.
• Multi-Select Behavior: On multi-select questions, models exhibited a precision-recall asymmetry. They tended to recover the single most salient correct option (high precision) but failed to enumerate the full set of valid interpretations (low recall). This suggests models struggle with the breadth of analytical perspective required for expert-level reasoning.
• Key Frame Selection: Adaptive key frame selection methods provided little to no performance gain, indicating that the bottleneck lies in stylistic vocabulary and cultural priors rather than temporal localization or frame selection.
• Position Bias: Open-source MLLMs exhibited a pronounced bias toward selecting the first option (A) in questions with five or more choices, whereas proprietary models showed significantly less bias.

Significance

The paper argues that the current trajectory of scaling generic video understanding is insufficient for achieving authentic creative domain understanding. The significant performance gap and specific failure patterns (e.g., inability to handle interpretive plurality, weakness in game arts) suggest that future progress requires:

• Richer Artistic Supervision: Integrating more diverse cultural and stylistic supervision into training pipelines.
• Multi-Faceted Evaluation: Moving beyond single-choice accuracy to metrics that capture partial credit and the breadth of reasoning.
• Domain-Specific Reasoning: Treating audiovisual arts as a distinct domain requiring specialized knowledge beyond general video comprehension.

MuseBench is released to facilitate continued progress toward MLLMs capable of genuine intent-level reasoning in the audiovisual arts.