MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos

1. Problem Statement

While Multimodal Large Language Models (MLLMs) have demonstrated strong capabilities in isolated visual or audio tasks, their ability to perform joint reasoning across omni-modal signals (visual, audio, and text) in long, complex, real-world videos remains largely unexplored.

• Current Limitations: Existing benchmarks often focus on short clips, ignore audio, treat audio as auxiliary, or rely on single-modality reasoning. They fail to capture the challenges of long-term temporal dependencies and the tight coupling required between audio and visual streams in real-world scenarios.
• The Gap: There is a lack of large-scale, systematic evaluation frameworks that test whether models can integrate evidence across modalities and time to solve complex reasoning problems, leading to a gap between model performance and human-level understanding.

2. Methodology: The MMOU Benchmark

The authors introduce MMOU, a comprehensive benchmark designed to evaluate advanced omni-modal perception and reasoning under realistic conditions.

Dataset Construction

• Scale: 15,000 carefully curated multiple-choice questions (MCQs) paired with 9,038 real-world web-collected videos.
• Video Characteristics:
  • Duration: Long-form videos ranging from 7 seconds to over 2 hours (Average: ~711 seconds / 12 minutes).
  • Domains: 10 major categories (e.g., Academic Lectures, Sports, News, Music, Daily Life) and 36 fine-grained subcategories.
  • Content: Naturally occurring, unscripted content with strong temporal and semantic alignment between audio and visual streams.
• Annotation Process:
  • Human Experts: 11 professionally trained annotators (backgrounds in linguistics, STEM, journalism) created the data.
  • Workflow: Annotators watched full videos, generated open-ended questions requiring joint audio-visual understanding, and annotated evidence timestamps.
  • Distractor Generation: Questions were converted to MCQs with 10 options (1 correct + 9 distractors). Distractors were generated using LLMs (GPT-5.2) to include "hard" semantic distractors and out-of-context options. 13% of questions include "None of the above" to prevent pattern matching.
  • Quality Control: Rigorous filtering removed ambiguous or single-modality solvable questions.

Skill Taxonomy

MMOU evaluates 13 fundamental skill categories, all requiring the integration of audio and visual cues:

1. Temporal Understanding: Reasoning about event order and progression.
2. Sub-scene Understanding: Identifying specific segments within long videos.
3. Holistic Video Reasoning: Global comprehension of theme/objective.
4. Inference & Context: Deducing unstated intentions or causes.
5. Needle-in-the-Haystack: Locating specific moments in long contexts.
6. Referential Grounding: Linking audio references to visual entities (and vice versa).
7. Counting & Comparative Reasoning: Quantitative analysis of events.
8. Object Interaction Reasoning: Understanding actions on objects over time.
9. Audio-Visual Stitching: Reasoning about edited segments and narrative continuity.
10. Tracking Spurious Correlations: Identifying non-intuitive audio-visual evidence.
11. (Plus 3 other specific reasoning types detailed in the paper).

3. Key Contributions

1. MMOU Benchmark: The first large-scale benchmark specifically targeting long-form, omni-modal reasoning with 15k QA pairs and 9k+ videos, covering 13 distinct reasoning skills.
2. Rigorous Evaluation: A systematic evaluation of 20+ state-of-the-art models (both open-source and proprietary), including audio-visual MLLMs, vision-only, audio-only, and text-only baselines.
3. Deep Failure Analysis: Identification of systematic failure modes, particularly in long-horizon reasoning and temporal retention, revealing that current models struggle even with basic skills in long videos.
4. Open-Ended Evaluation: A complementary evaluation protocol using LLM-as-a-Judge (GPT-5 and a custom-trained Qwen 3.5) to assess free-form generation, revealing gaps between MCQ recognition and true understanding.

4. Experimental Results

The authors evaluated a wide range of models, including Gemini 2.5 Pro, Qwen3-Omni, OmniVinci, and various unimodal baselines.

• Overall Performance:

  • Human Baseline: 84.3% accuracy.
  • Best Closed-Source Model: Gemini 2.5 Pro achieved 64.2% accuracy.
  • Best Open-Source Model: MiniCPM-o 4.5 (or similar top-tier open model) reached 46.8%.
  • Significant Gap: Even the best models are ~20 points below human performance, indicating fundamental limitations in joint reasoning.

• Modality Dependency:

  • Vision-Only Models: Qwen3-VL-32B achieved only 44%, proving visual cues alone are insufficient.
  • Audio-Only Models: Audio Flamingo 3 achieved 17.7%, and Qwen3-Omni (audio-only) achieved 35.6%.
  • Text-Only Models: GPT-5.2 achieved 40.7%, confirming that language priors alone cannot solve these tasks.
  • Cascaded Models: Models that generate captions first and then reason (e.g., Qwen3-VL + GPT-5) performed poorly (~33%), indicating that end-to-end joint perception is necessary.

• Skill-Specific Findings:

  • Models perform worst on Counting, Needle-in-the-Haystack, and Temporal Understanding.
  • Temporal Sensitivity: Accuracy drops significantly as the evidence for the answer appears later in the video (e.g., >30 minutes), highlighting a "lost in the middle" or context retention issue.

• Open-Ended vs. MCQ:

  • Open-ended evaluation scores were generally lower than MCQ scores.
  • Models often "know" the answer (high MCQ accuracy) but fail to articulate it correctly in free-form generation (low Correctness/Completeness in open-ended), suggesting MCQs may overestimate true understanding.

5. Significance and Impact

• Diagnostic Tool: MMOU provides a critical diagnostic framework to identify where and why current MLLMs fail, specifically in long-context, multi-modal reasoning.
• Research Direction: The results highlight that scaling alone is insufficient; future models need architectures capable of tight audio-visual fusion and long-horizon temporal reasoning.
• Real-World Applicability: By focusing on long, complex, real-world videos, MMOU bridges the gap between academic benchmarks and practical applications in education, media analysis, and human-computer interaction.
• Safety & Reliability: The benchmark exposes systematic failure modes, emphasizing the need for rigorous evaluation before deploying AI in high-stakes scenarios where misinterpretation of long-form content could have real-world consequences.

In conclusion, MMOU establishes a new standard for evaluating omni-modal intelligence, demonstrating that despite rapid progress, current models are still far from achieving human-level reasoning in complex, multi-modal, long-duration video environments.