LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMs

To address the gap in evaluating omnimodal large language models on long-form content, this paper introduces LVOmniBench, a new benchmark comprising 275 videos (10–90 minutes) and 1,014 QA pairs that reveals current models struggle with extended audio-visual comprehension, achieving below 35% accuracy for open-source models and up to 65% for Gemini 3 Pro.

The Gist

Imagine you are trying to teach a robot how to understand the world. So far, we've taught it to look at a single photo or listen to a 10-second clip of a song. It's doing pretty well at those short tasks. But real life isn't a series of snapshots; it's a continuous, flowing movie that can last for hours, filled with people talking, music playing, and things happening in the background.

This paper introduces LVOmniBench, a new "final exam" designed to test if our smartest AI robots can actually handle these long, complex movies.

Here is the breakdown of what they did, using some simple analogies:

1. The Problem: The "Short-Clip" Trap

Until now, most AI tests were like showing a student a 10-second video of a cat jumping and asking, "Did the cat jump?" The AI could easily memorize that specific jump.

But in the real world, videos are like 90-minute documentaries or vlogs. They have:

• Long-term memory: "What did that guy say 20 minutes ago?"
• Complex timing: "When did the music stop and the rain start?"
• Mixed signals: People talking over music while doing something with their hands.

Current AI models are like students who can memorize flashcards but fail a 3-hour final exam. They get lost when the story gets too long.

2. The Solution: LVOmniBench (The "Marathon" Test)

The researchers built a new test called LVOmniBench. Think of it as a marathon instead of a sprint.

• The Course: They collected 275 real-world videos (like cooking shows, travel vlogs, and interviews) that are 10 to 90 minutes long.
• The Questions: They wrote over 1,000 questions that require the AI to use both its eyes (video) and ears (audio) at the same time.
  • Example: "The man in the video says he is looking for 'Toby.' How many times does he actually see Toby in the yard?"
  • Why it's hard: The AI has to listen to the name, watch the video to find the dog, count the sightings, and ignore the times the dog isn't there.

3. The Results: The "Rich Kid" vs. The "Struggling Student"

The researchers tested the best AI models on this marathon. The results were a bit shocking:

• The "Proprietary" Models (The Rich Kids): These are the super-expensive, closed-source models like Gemini 3 Pro. They are like students with private tutors and unlimited study time. They scored around 65%. They did well, but they still got about a third of the questions wrong.
• The "Open-Source" Models (The Struggling Students): These are the free, community-built models. They scored below 35%.
  • The Metaphor: If you guess randomly on a multiple-choice test, you'd get 25% right. These open-source models were barely doing better than random guessing. They got completely lost in the long videos.

4. Where Did They Fail? (The "Hallucination" Zone)

The paper found that the AI fails in three specific ways, like a student who is bad at:

• Counting: "How many times did the dog bark?" (The AI often loses count).
• Music Perception: "What kind of instrument is playing?" (The AI hears noise but can't identify the melody).
• Time Travel: "What happened 15 minutes ago?" (The AI forgets the beginning of the video by the time it reaches the end).

5. The Big Takeaway

The main message is that we are not there yet.

While AI is amazing at looking at a picture or listening to a short sentence, it is currently terrible at understanding a long, continuous story where sound and sight are mixed together. The "open-source" models are essentially blind and deaf to the nuances of long videos.

Why does this matter?
Because the future of AI isn't just about answering questions about a photo. It's about having an AI assistant that can watch a 2-hour security camera feed to find a thief, or listen to a 45-minute lecture and summarize the key points. This new test (LVOmniBench) is the first step in forcing AI developers to build models that can actually handle the length and complexity of real life.

In short: We built a harder test, and the AI failed it. Now, we know exactly what to fix to make the next generation of AI truly "omni-smart."

Technical Summary

1. Problem Statement

Recent advancements in Omnimodal Large Language Models (OmniLLMs) have improved the ability to process text, images, audio, and video simultaneously. However, current evaluation benchmarks suffer from two critical limitations:

• Short Duration: Existing benchmarks primarily focus on short clips (10 seconds to 5 minutes), failing to reflect real-world scenarios where videos often span tens of minutes.
• Modality Isolation: Most benchmarks evaluate static image-audio pairs or short video clips, lacking rigorous tests for long-form joint audio-visual reasoning.
• Gap in Capability: While models perform well on isolated tasks, they struggle with the complex temporal dependencies, fine-grained cross-modal alignment, and long-term memory retention required for extended audio-visual content.

2. Methodology: LVOmniBench Construction

The authors introduce LVOmniBench, the first comprehensive benchmark designed specifically for evaluating OmniLLMs on long-form audio-visual content. The construction process involved a rigorous, multi-stage pipeline:

A. Data Collection & Curation

• Source: Videos were sourced from YouTube, strictly adhering to Creative Commons licenses to ensure open accessibility.
• Domains: Content spans five major categories (Entertainment, Lifestyle, DIY & Cooking, Record, Film & TV) and 21 fine-grained subcategories.
• Selection Criteria:
  • Duration: Videos range from 10 to 90 minutes (average ~34.5 minutes), representing a 6x increase in temporal scale compared to existing benchmarks.
  • Quality: Only videos with synchronized audio, rich dynamics, and sufficient information density were selected.
  • Final Dataset: 275 high-quality videos totaling 140 hours of content.

B. Question Annotation & Refinement

• Manual Curation: 1,014 multiple-choice questions (QA pairs) were manually created by human experts.
• Question Types: Questions are categorized into four cognitive dimensions:
  1. Perception: Extracting attributes, counting, and music perception.
  2. Understanding: Entity recognition, event understanding, and human-centric modeling.
  3. Inference: Sound, spatial, and temporal inference requiring cross-modal alignment.
  4. Logical Reasoning: Multi-step reasoning, causal tracking, and complex deduction.
• Difficulty Stratification: Each question is labeled as Low, Medium, or High difficulty based on perceptual granularity, temporal span, and inference complexity.
• Quality Control:
  • Unimodal Bias Check: Questions solvable by only audio or only video were discarded.
  • Hallucination Prevention: Distractors were derived strictly from the video/audio content, not common sense.
  • Timestamp Removal: Explicit timestamps were minimized to prevent trivial shortcuts.

3. Key Contributions

1. First Long-Form Omni-Benchmark: LVOmniBench is the first dataset to rigorously evaluate OmniLLMs on videos averaging 34 minutes (vs. ~2-5 minutes in prior works).
2. Hierarchical Evaluation: The dataset introduces a multi-dimensional evaluation framework covering perception, understanding, inference, and logical reasoning, with explicit difficulty levels.
3. Rigorous Manual Annotation: Unlike many benchmarks that rely on automated LLM generation (prone to hallucinations), LVOmniBench uses strict human-in-the-loop annotation to ensure high-fidelity, unimodal-bias-free QA pairs.
4. Comprehensive Baseline Analysis: The paper provides extensive evaluations of both proprietary (Gemini 3 Pro/Flash) and open-source models, revealing significant performance gaps.

4. Experimental Results

The authors evaluated state-of-the-art models, including Gemini 3 Pro/Flash, Ming-Flash-Omni, MiniCPM-o, Qwen3-Omni, and VideoLLaMA2.

• Overall Performance:
  • Proprietary Models: Gemini 3 Pro achieved the highest accuracy at 65.8%, followed by Gemini 3 Flash at 59.0%.
  • Open-Source Models: All open-source models struggled significantly, with accuracies generally below 35% (e.g., Qwen3-Omni at 35.8%, Ming-Omni at 34.6%). This is often only marginally better than random chance.
• Task-Specific Challenges:
  • Counting & Music Perception: These were the most difficult tasks for all models, even for Gemini 3 Pro.
  • Audio Dependency: Models relying solely on video (visual-only) or audio (ASR-only) performed significantly worse than joint A+V models, proving the necessity of true multimodal fusion.
  • Long-Context Degradation: Performance dropped sharply as video duration increased, highlighting a bottleneck in processing extended temporal sequences.
• Error Analysis:
  • Perception Errors (34%): Models often miscount objects or fail to perceive non-verbal audio cues (tone, emotion, music).
  • Localization Errors (19.6%): Models struggle to pinpoint the exact temporal occurrence of events in long videos.
  • Cross-Modal Semantic Gap (19%): Models frequently fail to align audio and visual cues, often relying on one modality while ignoring the other.
  • Reasoning Errors (23.5%): Complex logical deduction and spatial reasoning remain weak points.

5. Significance and Future Directions

• Benchmarking Standard: LVOmniBench sets a new standard for evaluating the "real-world" capabilities of OmniLLMs, moving beyond short clips to complex, long-form scenarios.
• Identifying Bottlenecks: The results clearly indicate that while proprietary models have made strides, open-source models lack the architecture and training data to handle long audio-visual sequences effectively.
• Research Directions:
  • Long-Context Modeling: Developing better token compression and memory mechanisms specifically for long audio sequences (which have higher information density than video).
  • Cross-Modal Alignment: Improving the ability to fuse audio and visual features at a fine-grained level to reduce modality bias.
  • Temporal Grounding: Enhancing the ability to locate events precisely within extended timelines.

In conclusion, LVOmniBench exposes a critical gap in current OmniLLM capabilities regarding long-form audio-visual understanding. It serves as a catalyst for developing next-generation models capable of robust, real-world multimodal reasoning.