WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMs

The paper introduces WorldSense, the first benchmark designed to evaluate multimodal LLMs' real-world understanding by integrating synchronized audio, visual, and text inputs across diverse scenarios, revealing that current state-of-the-art models still face significant challenges in achieving coherent omni-modal perception.

The Gist

Imagine you are trying to teach a robot how to drive a car. You could show it pictures of the road (vision) and tell it the rules of the road (text). But what if you only showed it pictures? It would miss the sound of a siren approaching from behind or the honk of a car in its blind spot. It would be like driving with your eyes open but your ears plugged.

This is the problem the paper WorldSense is trying to solve.

Here is a simple breakdown of what the researchers did, using some everyday analogies:

1. The Problem: The "One-Eared" Robot

For a long time, AI models (the "brains" of robots) have been great at looking at pictures and reading text. But when it comes to video, they often ignore the sound. They are like a person watching a movie with the volume turned all the way down. They might see a character crying, but they won't hear the sad music that tells you why they are crying, or the specific tone of their voice that reveals they are actually angry, not sad.

Existing tests for these AI models were like asking a blindfolded person to identify a song just by looking at the album cover. It doesn't test if they can really understand the world.

2. The Solution: The "WorldSense" Exam

The researchers created a new, super-challenging test called WorldSense. Think of this as the "Bar Exam" for AI, but specifically for understanding real life.

• The Ingredients: They gathered 1,662 short video clips from the real world. These aren't just cartoons; they are real people doing real things, with all the background noise, music, and conversations intact.
• The Questions: They wrote over 3,000 multiple-choice questions about these videos.
• The Catch: The questions are designed so that you cannot answer them correctly if you ignore either the picture or the sound.

Example from the paper:

• The Scene: A man is holding a fruit.
• The Question: "What is he doing?"
• The Trap: If you only look at the video, you see a man holding a blueberry. You might guess he is "showing the color." But if you listen to the audio, you hear him say, "This one is bigger than a quarter." Now you know he is showing the size.
• The Result: An AI that ignores sound will fail. An AI that ignores the video will also fail. It needs to use both to get it right.

3. The Results: The AI is Still a Toddler

The researchers took the smartest AI models available (including big names like Google's Gemini and OpenAI's GPT-4o) and gave them this exam.

• The Score: The best AI in the world only got about 65% of the answers right.
• The Reality Check: Many open-source (free) AI models scored around 25%. Since there are usually 4 options in a multiple-choice question, 25% is basically random guessing.

The Analogy: Imagine a human taking a driving test. If they got 65%, they would pass, but they'd be a risky driver. If they got 25%, they would fail immediately and be told to go back to driving school. The paper shows that even the "smartest" AIs are currently terrible at understanding the full picture of real life.

4. Why Are They Failing?

The paper found three main reasons why the AIs are struggling:

1. They are "Deaf": They are great at seeing but terrible at hearing. They often miss the tone of voice, the background noise, or the music.
2. They Don't Mix Well: Even when they do hear and see, they don't know how to combine the two. It's like having a brain that processes eyes and ears in two separate rooms that never talk to each other.
3. They Can't Reason: Sometimes they see the right thing and hear the right thing, but they can't put the pieces together to make a logical conclusion.

5. The Future: Building a "Super-Sense"

The authors hope that by using WorldSense as a standard test, other researchers will be forced to build better AI. They want to create models that don't just "see" and "read," but truly perceive the world like humans do—integrating sight, sound, and context all at once.

In a nutshell:
The world is a noisy, colorful, complex place. Current AI is like a tourist who only reads the guidebook and ignores the sights and sounds around them. WorldSense is the test that forces the AI to step outside, listen to the street, and actually understand what's happening. Right now, the AI is failing the test, but this new benchmark gives us a roadmap to fix it.

Technical Summary

1. Problem Statement

Current Multimodal Large Language Models (MLLMs) have made significant progress in visual-language tasks (e.g., image captioning, visual question answering). However, real-world understanding requires the integration of omnimodal inputs, specifically the synergistic combination of visual, auditory, and textual information.

Existing benchmarks suffer from critical limitations:

• Modality Gap: Most benchmarks focus primarily on visual-text pairs, neglecting audio or treating it as a secondary, decoupled signal.
• Weak Coupling: Existing audio-visual benchmarks often rely on static images or simple scenarios where audio and video are not tightly coupled. In many cases, a model can answer correctly using only one modality, failing to test true multimodal reasoning.
• Lack of Diversity: Current datasets often lack fine-grained subcategories, cover limited domains, or suffer from low-quality annotations and monotonous question patterns.
• Evaluation Deficit: There is no comprehensive benchmark to assess an MLLM's ability to reason about complex, dynamic real-world scenarios where audio and visual cues are interdependent.

2. Methodology: The WorldSense Benchmark

The authors introduce WorldSense, the first benchmark designed to evaluate MLLMs on omnimodal video understanding (simultaneous processing of video, audio, and text).

A. Design Principles

1. Comprehensive Domain Coverage: Videos are categorized into 8 primary domains (e.g., Tech & Science, Culture & Politics, Daily Life, Film & TV, Performance, Games, Sports, Music) and 67 fine-grained subcategories to ensure ecological validity.
2. Diverse Acoustic Modalities: The dataset includes speech, environmental sounds, and music, ranging from linguistic content to non-verbal and abstract auditory cues.
3. Multilevel Cognitive Assessment: Questions are designed across three cognitive levels:
   • Recognition: Detecting basic elements.
   • Understanding: Comprehending multimodal relationships.
   • Reasoning: High-level inference (causal, abstract).

B. Data Collection & Curation

• Source: Primarily sourced from FineVideo (high-quality YouTube videos with strong audio-visual correlations) and supplemented with MusicAVQA for musical content.
• Filtering: An initial pool of ~8,000 clips was filtered based on predefined taxonomies, pre-computed audio-visual correlation metrics, and dynamic content scores.
• Final Dataset: 1,662 high-quality, synchronized audio-visual video clips with an average duration of 141.1 seconds.

C. Annotation Protocol

• QA Generation: 3,172 multiple-choice question-answer (QA) pairs were created across 26 distinct tasks.
• Human-in-the-Loop: 80 professional annotators manually labeled the data.
• Quality Control: A rigorous dual-verification system was employed:
  1. Expert Review: Checked for linguistic clarity, multimodal necessity (ensuring the answer requires both audio and video), and difficulty.
  2. Automated Verification: MLLMs (e.g., Qwen2-VL, Video-LLaMA2) were used to verify that questions require multiple modalities. If a model answered correctly using only one modality, the question was flagged for revision.

D. Evaluation Paradigm

Models are evaluated on accuracy in selecting the correct option from multiple choices. The evaluation includes ablation studies to measure performance under different input configurations:

• Video + Audio (Full)
• Video + Subtitles (Textual proxy for audio)
• Video Only
• Audio Only

3. Key Contributions

1. WorldSense Benchmark: The first comprehensive benchmark for real-world omnimodal video understanding, featuring tightly coupled audio-visual inputs, diverse content (8 domains, 67 subcategories), and high-quality, manually verified annotations.
2. Extensive Evaluation: A systematic evaluation of a broad spectrum of MLLMs, including open-source video-audio models (e.g., Unified-IO-2, VideoLLaMA2), open-source video-only models (e.g., Qwen2-VL, LLaVA-Video), and proprietary systems (e.g., GPT-4o, Gemini 1.5/2.5 Pro).
3. Insightful Analysis: Identification of critical gaps in current models, specifically the inability to effectively integrate modalities and the reliance on visual cues over raw audio.

4. Experimental Results

Overall Performance

• State-of-the-Art Limitations: Even the best-performing proprietary model, Gemini 2.5 Pro, achieved only 65.1% accuracy.
• Open-Source Struggles: Most open-source video-audio models performed near random guessing (~25% accuracy), significantly underperforming compared to video-only models. This suggests that simply adding an audio encoder without sophisticated integration mechanisms does not improve performance.
• Modality Dependency: Models generally perform better with visual inputs. However, removing audio causes a significant drop in performance for tasks requiring auditory cues, highlighting the necessity of integrated processing.

Detailed Findings

• Audio Understanding Gap: Models consistently underperform on audio-specific tasks (e.g., counting sounds, identifying music genres, recognizing emotional tone) compared to visual tasks.
• Raw Audio vs. Subtitles:
  • Providing subtitles to video-only models improves performance significantly (e.g., +8.8% for Qwen2-VL), proving the value of textual audio proxies.
  • However, raw audio provides further gains over subtitles (e.g., Gemini 1.5 Pro improved from 39.3% with subtitles to 48.0% with raw audio). This indicates that paralinguistic cues (prosody, intonation, environmental sounds) contain information subtitles cannot capture.
• Failure Modes: Error analysis revealed three main failure types:
  1. Audio Understanding Errors: Misinterpreting sound types or rhythms.
  2. Visual Understanding Errors: Missing fine-grained visual details.
  3. Reasoning Errors: Correctly perceiving inputs but failing to logically infer the answer.

5. Significance and Future Directions

Significance:
WorldSense establishes a new standard for evaluating MLLMs, moving beyond static image-text pairs to dynamic, real-world scenarios requiring synergistic perception. The results expose a significant "omnimodal reasoning gap," indicating that current models are not yet ready for complex, embodied AI applications where audio-visual integration is critical (e.g., autonomous driving, assistive robotics).

Future Directions Proposed:

1. Coupled Multimodal Training: Utilizing naturally interleaved audio-visual-text data to teach models cross-modal dependencies.
2. Architectural Improvements: Shifting from late-fusion (processing streams separately) to early-fusion mechanisms with enhanced attention layers for deep integration.
3. Advanced Alignment: Developing progressive alignment strategies to better synchronize information across modalities.
4. Reasoning Strengthening: Incorporating diverse reasoning-focused data to improve logical inference capabilities beyond simple pattern matching.

In conclusion, WorldSense provides a critical platform for diagnosing the limitations of current MLLMs and guiding the development of next-generation models capable of true human-like, omnimodal world understanding.