Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalities

The paper introduces Daily-Omni, a comprehensive audio-visual benchmark with 1,197 questions designed to evaluate cross-modal temporal reasoning, revealing that current multimodal large language models still struggle with alignment-critical tasks despite strong unimodal performance.

The Gist

Imagine you are watching a movie with the sound turned off. You see a character slam a door and look angry. You might guess they are mad. Now, imagine you have the sound on but the screen is black. You hear a loud CRASH and a shout. You might guess something broke.

But what if you need to know exactly when the shout happened relative to the crash? Did the shout come before the crash to warn them? Did it happen after to express frustration? Or did they happen at the exact same split second?

This is the specific problem the paper Daily-Omni is trying to solve. It's like a "driver's license test" for AI, but instead of driving a car, the AI has to drive a conversation between its eyes (video) and its ears (audio) at the exact same time.

Here is the breakdown of the paper in simple terms:

1. The Problem: The AI is "Deaf-Blind" in Time

Current super-smart AI models (called Multimodal Large Language Models) are great at looking at pictures or listening to music separately. But when you give them a video with sound, they often struggle to sync the two.

Think of it like a person trying to dance to a song while wearing noise-canceling headphones and blindfolds, then trying to guess the rhythm. They might know the steps (video) and the beat (audio), but they can't feel how they fit together in the moment. The paper argues that most AI today is bad at this "temporal alignment"—figuring out which sound matches which visual action at the exact same time.

2. The Solution: A New "Gym" for AI (Daily-Omni)

The researchers built a new testing ground called Daily-Omni.

• The Workout: They collected 684 real-world videos (like people cooking, fixing cars, or playing music) and created nearly 1,200 questions about them.
• The Challenge: The questions aren't just "What color is the car?" They are tricky, like: "Did the person drop the glass before or after the dog barked?" or "Why did the crowd cheer right when the singer hit that high note?"
• The Goal: To force the AI to stop guessing and actually connect the dots between what it sees and what it hears in real-time.

3. How They Built It: The "Smart Assembly Line"

Making these questions by hand would take forever. So, the team built a semi-automatic factory line:

1. The Scribes: They used powerful AI to write down what was happening in the video (visuals) and what was happening in the sound (audio) separately.
2. The Editors: Another AI checked to make sure the story made sense (e.g., if the visual says "a man is running," the audio shouldn't say "a cat is sleeping").
3. The Time-Keepers: This is the magic step. They taught the AI to link specific moments: "The sound of the door slamming happened at the exact same time the visual of the door closing appeared."
4. The Filter: They made sure the questions couldn't be answered just by reading the text. If a human could guess the answer without watching the video or listening to the sound, the question was thrown out.

4. The Results: The AI is Still Stumbling

They tested 24 different top-tier AI models on this new gym. The results were a bit of a reality check:

• The "Text-Only" Trap: Surprisingly, some AI models did almost as well just by reading the question text, without even looking at the video or listening to the audio. This means the questions were sometimes too easy to guess.
• The "Sync" Struggle: Even the best AI models (like the ones from Google and Alibaba) struggled with the "time-sync" questions. They often got the facts right but the timing wrong.
• The Simple Baseline Wins: The researchers built a simple, "training-free" tool called the Daily-Omni Agent. It's like a human taking notes: "First I see X, then I hear Y." Even though this tool is simple and not a super-complex neural network, it actually beat several of the massive, expensive AI models.

The Big Takeaway

The paper concludes that while AI is getting very good at seeing and hearing separately, it is still terrible at listening and watching at the same time.

The Metaphor:
Imagine a symphony orchestra. The current AI models are like musicians who can play their instruments perfectly on their own. But when they try to play together, they are out of sync. One is rushing, the other is dragging. Daily-Omni is the conductor trying to teach them how to play in perfect harmony.

The paper suggests that for AI to truly understand the real world (like a self-driving car hearing a siren while seeing a police car, or a robot understanding a human's tone of voice while watching their facial expression), we need to fix this "temporal alignment" problem first. Until then, our AI is still a bit out of step.

Technical Summary

Here is a detailed technical summary of the paper "Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalities".

1. Problem Statement

While Multimodal Large Language Models (MLLMs) have achieved significant success in unimodal tasks (visual or audio independently), their ability to process cross-modal information synchronously remains under-explored. Existing Audio-Visual (AV) benchmarks suffer from three critical limitations:

1. Domain Bias: Many focus on specialized scenarios (e.g., music or panoramic views) rather than diverse daily life.
2. Lack of Temporal Dynamics: Many use static image-audio pairs or short clips that ignore the complex temporal evolution inherent in real-world videos.
3. Narrow Scope: Tasks often focus on simple captioning or open-ended responses rather than rigorous reasoning requiring precise temporal alignment between audio and visual events.

Current models often struggle to align specific sounds with visual actions occurring at the exact same moment, a capability crucial for understanding real-world environments.

2. Methodology

A. The Daily-Omni Benchmark

The authors introduce Daily-Omni, a high-quality, multiple-choice AV Question Answering (QA) benchmark designed to evaluate temporally aligned reasoning.

• Dataset Composition: 684 real-world videos (sourced from AudioSet, Video-MME, and FineVideo) spanning 11 YouTube categories. Videos are segmented into 30s and 60s clips.
• Scale: 1,197 multiple-choice questions across 6 task families:
  1. AV Event Alignment: Identifying simultaneous audio-visual events.
  2. Event Sequence: Determining the temporal order of events.
  3. Reasoning: Explaining causes of events.
  4. Inference: Speculating on implicit information.
  5. Comparative: Comparing audio-visual info across segments.
  6. Context Understanding: Analyzing surrounding context.
• Filtering: Questions requiring only single-modality knowledge (e.g., counting, celebrity ID) are excluded to ensure true multimodal reasoning is required.

B. Scalable QA Construction Pipeline

To address the high cost of manual annotation, the authors developed a semi-automatic pipeline:

1. Segmented Annotation: Videos are split into shorter segments (10s/20s) to improve MLLM annotation quality.
2. Independent Annotation: Visual and audio modalities are annotated separately using Gemini 2.0 Flash to avoid cross-modal hallucinations.
3. Consistency & Revision:
   • Visual: The full video is reviewed to ensure character/object consistency across segments.
   • Audio: A Reasoning LLM (Deepseek-R1) uses visual context to correct audio misidentifications (e.g., changing "generic impact" to "door slam" based on visual evidence).
4. Temporal Alignment Elicitation: The model is prompted to explicitly identify which visual events occur concurrently with specific audio events, creating aligned event pairs.
5. QA Generation & Optimization: Deepseek-R1 generates questions and distractors.
6. Leakage Filtering: A critical step where powerful LLMs (GPT-4o, Deepseek-V3) attempt to answer questions using text-only input. If they succeed, the question is discarded to prevent models from bypassing audio-visual processing.
7. Human Verification: Final manual review ensures unambiguous answers and true multimodal necessity.

C. Diagnostic Baseline: Daily-Omni Agent

To serve as a reference for "ideal" performance without end-to-end training, the authors built a training-free modular baseline:

• Architecture: Composes off-the-shelf unimodal models (Qwen2-Audio, Qwen2.5-VL, Whisper-Large-V2, and Qwen2.5-14B-Instruct).
• Workflow:
  1. Generates independent visual and audio annotations.
  2. Performs consistency checks.
  3. Explicit Alignment: Uses a video temporal grounding model to localize critical events and retrieve corresponding audio descriptions, creating explicit aligned event pairs.
  4. Feeds these structured, aligned inputs to a reasoning LLM to generate the final answer.

3. Key Contributions

1. Daily-Omni Benchmark: A rigorous dataset with 1,197 QA pairs focusing on complex, temporally aligned cross-modal reasoning in daily scenarios.
2. Scalable Pipeline: An automated framework that reduces human labor (requiring only ~30 hours for a single annotator to filter 1,197 questions) while maintaining high quality through iterative revision and leakage filtering.
3. Diagnostic Framework: A comprehensive evaluation suite testing 24 models under 37 settings (Audio+Video, Audio-only, Video-only, Text-only) and a modular baseline to isolate temporal alignment capabilities.

4. Results & Findings

The authors evaluated 24 foundation models (including Qwen Omni series, Gemini, Unified-IO, and VideoLLaMA) against the benchmark:

• Temporal Alignment is the Bottleneck: Many end-to-end MLLMs struggle significantly with alignment-critical questions. Surprisingly, the training-free Daily-Omni Agent (61.82% accuracy) outperforms several recent open-source unified models (e.g., Ola at 50.71%, Unified-IO 2 at ~28%).
• Modality Dependence:
  • Removing either audio or visual input causes a drastic performance drop (e.g., Gemini 2.5 Flash drops from 73.06% to 44.61% without audio).
  • Unlike many video benchmarks where audio is redundant, Daily-Omni shows that audio is indispensable; audio-only performance often exceeds visual-only performance, highlighting the richness of acoustic events in daily life.
• Model Performance:
  • Top Performers: Proprietary models like Gemini 2.5 Flash (73.06%) and open-source Qwen3-Omni (71.85%) lead the pack, likely due to better temporal modeling components (e.g., TMRoPE) and training data.
  • Weaknesses: Older or smaller unified models often fail to synthesize cross-modal information, performing worse than text-only baselines on specific tasks.
• Impact of Alignment: In ablation studies, the "Smart Alignment" strategy (explicitly identifying and aligning event pairs) significantly boosted the baseline's performance compared to "Naive" or "No Alignment" approaches, proving that explicit temporal grounding is key to solving these tasks.

5. Significance

• Benchmarking Standard: Daily-Omni fills a gap in the literature by providing a standardized, scalable, and rigorous testbed for temporal cross-modal reasoning, moving beyond simple event detection.
• Diagnostic Insight: The results indicate that current unified MLLM architectures lack robust mechanisms for spatio-temporal alignment. The fact that a modular, training-free baseline outperforms unified models suggests that simply scaling parameters is insufficient; explicit alignment strategies are necessary.
• Future Direction: The paper advocates for future research to prioritize accurate multimodal temporal grounding and alignment mechanisms to achieve true omnimodal understanding capable of interacting with the physical world.