OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs

This paper introduces OmniVideoBench, a large-scale benchmark comprising 1,000 rigorously annotated QA pairs derived from 628 diverse videos, designed to comprehensively evaluate the synergistic audio-visual reasoning capabilities of multimodal large language models and highlight the significant performance gap between current models and human reasoning.

The Gist

Imagine you are teaching a robot to understand the world not just by looking at pictures, but by watching movies and listening to the soundtrack simultaneously. That is essentially what this paper, OmniVideoBench, is all about.

Here is the story of the paper, broken down into simple concepts and everyday analogies.

1. The Problem: The "Blind" and the "Deaf" Robot

For a long time, AI models (robots) have been great at looking at images and reading text. But when it comes to videos, they often struggle.

Think of existing AI tests like a driving test where you are only allowed to look at the road, but you are forbidden from listening to the engine, the horn, or the radio.

• The Issue: Current tests for video AI often ignore the sound or treat it as an afterthought. They ask questions like, "What color is the car?" (Visual) but ignore, "Why is the car speeding?" (Audio: The engine revving).
• The Result: The AI might guess the right answer by looking, but it doesn't truly understand the scene. It's like a student who memorizes the answer key without understanding the lesson.

2. The Solution: The "OmniVideoBench" Exam

The researchers (the NJU-LINK Team) built a new, super-challenging exam called OmniVideoBench.

• The Classroom: They collected 628 real-world videos (like news clips, sports games, vlogs, and documentaries) ranging from a few seconds to 30 minutes long.
• The Test Questions: They wrote 1,000 questions about these videos. But these aren't simple "What is this?" questions. They are complex puzzles that require listening and watching together.
  • Example: "The poster says 'No One Fights Alone,' but the person in the video is hiding in a corner. What does this imply about their relationship?"
  • To answer this, the AI must see the poster (Visual) and hear the dialogue or music (Audio) to understand the emotional context.
• The "Reasoning Trace": This is the coolest part. The researchers didn't just write the right answer; they wrote out the step-by-step thought process a human would use.
  • Step 1: Listen to the voice saying "I deployed the bomb."
  • Step 2: Look at the wall to see where the poster is.
  • Step 3: Connect the two to figure out the location.
  • This forces the AI to show its work, proving it didn't just guess.

3. The Results: The AI is Still in Kindergarten

The researchers tested the world's smartest AI models (like Google's Gemini and various open-source models) on this new exam.

• The Score: The results were humbling. Even the best AI models scored around 59%.
  • Analogy: If the passing grade is 60%, the smartest AI in the world is currently failing the test.
• The Gap: Humans scored 82%. There is a huge gap between how humans understand a movie and how AI does.
• The Weak Spots:
  • Music: AI is terrible at understanding music. If a video has a sad song playing, the AI often misses the emotion. It's like watching a sad movie with the volume off; the AI sees the crying but doesn't "feel" the sadness.
  • Long Videos: AI gets lost in long movies (over 10 minutes). It's like trying to remember every detail of a 3-hour movie after watching it once; the AI forgets the beginning by the time it gets to the end.
  • Open-Source vs. Closed-Source: The "secret sauce" models (like Google's) did better than the public ones, but even they struggled.

4. Why This Matters

Think of OmniVideoBench as a new, stricter driver's license test.

• Before, the test only asked, "Can you steer the car?"
• Now, the test asks, "Can you steer the car while listening to the GPS, hearing the engine sputter, and noticing the pedestrian waving at you?"

The paper concludes that while AI is getting better, it still lacks true common sense when it comes to combining sight and sound. The researchers are releasing this "exam" to the public so that other scientists can use it to train their robots to become smarter, more attentive, and better at understanding the messy, noisy, beautiful real world.

In a nutshell: We built a tough test to see if AI can really "watch a movie" with its ears and eyes open. Currently, the AI is still a bit clumsy, but this test will help it learn to dance.

Technical Summary

1. Problem Statement

While Multimodal Large Language Models (MLLMs) have advanced significantly in image-text and video-text understanding, synergistic audio-visual reasoning remains underdeveloped. Existing benchmarks suffer from three critical limitations:

• Modality Neglect: Many benchmarks focus primarily on visual cues, treating audio as auxiliary or optional, failing to test true cross-modal integration.
• Logical Inconsistency: Some benchmarks integrate audio and video in ways that do not require logical synergy (e.g., questions solvable by vision alone).
• Temporal Limitations: Most existing datasets focus on short clips, failing to capture long-term temporal dependencies and dynamic scene transitions essential for comprehensive video understanding.

Consequently, current evaluations do not accurately reflect a model's ability to perform robust inference where audio and vision must be logically and consistently integrated.

2. Methodology: OmniVideoBench Construction

The authors introduce OmniVideoBench, a large-scale, rigorously designed benchmark specifically for evaluating audio-visual collaborative reasoning.

Data Collection

• Source: 628 real-world videos sourced from YouTube and Bilibili.
• Diversity: Covers 8 major categories (Vlog, News, Cartoon, Sports, Documentary, TV, Ego, Others) and 68 subcategories.
• Duration: Videos range from several seconds to 30 minutes, ensuring evaluation across varying temporal scales.
• Quality Control: Videos are restricted to recent publications (post-June 2024) to prevent data leakage from training sets. They must contain rich, complementary audio and visual information (no subtitles, no redundant audio).

Annotation & QA Design

• Scale: 1,000 high-quality Question-Answer (QA) pairs.
• Task Types: 13 distinct reasoning categories, including:
  • Perception: Fine-grained perception, counting, spatial reasoning.
  • Reasoning: Causal inference, relationship reasoning, hypothetical reasoning, ego reasoning.
  • Understanding: Temporal understanding, summarization, sentiment analysis, background & music understanding.
• Reasoning Traces: Each QA pair is annotated with step-by-step reasoning chains. Each step explicitly identifies the modality (Audio/Visual), the evidence (specific content), and the inference. This allows for granular analysis of how a model reasons, not just the final answer.
• Constraints:
  • Answers must be unique and correct.
  • Questions must not be solvable by a single modality (verified via filtering).
  • Distractors must be relevant and semantically balanced to prevent bias.

Quality Assurance Pipeline

A multi-stage filtering process ensures data integrity:

1. Unimodal Filtering: An advanced MLLM (Gemini 2.0 Flash) filters out questions solvable using only visual or only audio input.
2. Textual Bias Filtering: A strong LLM (DeepSeek-V3.1) removes questions answerable via common sense or textual cues in the prompt/options.
3. Manual Refinement: Human annotators verify correctness, uniqueness, and enrich the data with atomic reasoning steps.

3. Key Contributions

1. First Large-Scale Synergistic Benchmark: OmniVideoBench is the first benchmark to comprehensively evaluate synergistic audio-visual reasoning with a focus on modality complementarity and logical consistency across long-form videos.
2. Step-by-Step Reasoning Annotations: Unlike standard benchmarks, it provides explicit reasoning traces (Modality + Evidence + Inference) for every question, enabling fine-grained error analysis.
3. Comprehensive Taxonomy: It covers 13 diverse task types and 68 subcategories, addressing gaps in music understanding, long-context reasoning, and cross-modal causal inference.
4. Rigorous Filtering: The dataset strictly eliminates questions solvable by single modalities or textual cues, ensuring that high scores genuinely reflect audio-visual understanding.

4. Experimental Results

The authors evaluated a wide range of closed-source (e.g., Gemini-2.5-Pro, Gemini-2.0-Flash) and open-source (e.g., Qwen3-Omni, Qwen2.5-Omni, VideoLLaMA2) models.

Key Findings

• Significant Performance Gap: No model achieved a passing score (<60%). The best performer, Gemini-2.5-Pro, reached only 58.90%. Open-source models generally performed near random guessing (e.g., Qwen3-Omni-30B-A3B at ~38%).
• Human vs. Model Gap: Human annotators achieved 82.69% accuracy, highlighting a substantial gap between current MLLMs and human-level reasoning.
• Audio Type Sensitivity: Models struggle significantly with music.
  • Gemini-2.5-Pro: 38.46% accuracy on music vs. 61.66% on speech.
  • This suggests current models cannot effectively translate abstract emotional/atmospheric cues in music into reasoning.
• Long-Video Challenges: Performance degrades on videos longer than 10 minutes. While Gemini-2.5-Pro shows robustness, many models (e.g., Gemini-2.0-Flash) fail to maintain coherence over long contexts.
• Task Difficulty:
  • Hardest: Background & Music Understanding (<50% even for top models).
  • Easiest: Relationship Reasoning and Summarization (>80% for top models), likely because these rely more on explicit language recognition than abstract cross-modal abstraction.
• ASR Limitations: Providing Automatic Speech Recognition (ASR) transcripts to visual-only models improves performance but does not replace the need for raw audio processing, especially for non-speech sounds (music, ambient noise).
• Open-Ended vs. MCQ: Models perform significantly worse on open-ended QA compared to Multiple Choice Questions (MCQ), indicating that MCQ formats may overestimate capabilities by allowing random guessing or exploiting option biases.

5. Significance

• Benchmarking Standard: OmniVideoBench sets a new standard for evaluating "Omni-modal" capabilities, moving beyond simple perception to complex, multi-step reasoning.
• Research Direction: The results highlight specific bottlenecks for future research:
  • Improving music and sound understanding (low-semantic acoustic cues).
  • Enhancing long-context temporal reasoning.
  • Developing native cross-modal architectures that do not rely on ASR as a crutch.
• Open Science: The authors plan to release the dataset and code to foster the development of more generalizable and robust multimodal reasoning systems.

In conclusion, OmniVideoBench reveals that despite rapid progress, current MLLMs are far from achieving genuine, synergistic audio-visual understanding comparable to human cognition, particularly in complex, long-form, and music-rich scenarios.