Technical Summary: EgoAVU: Egocentric Audio-Visual Understanding

Problem Statement

Understanding egocentric videos is critical for embodied intelligence and mixed realities, yet it presents unique challenges due to highly dynamic camera motion, limited fields of view, and frequent self-occlusions. While recent Multimodal Large Language Models (MLLMs) can process both visual and auditory inputs, their ability to jointly understand audio-visual signals in egocentric settings remains under-explored.

The primary bottleneck is the lack of high-quality, large-scale data with coherent joint-modality information. Existing egocentric datasets (e.g., Ego4D, MultiHop-EgoQA) rely heavily on human narrations that often bias toward human-object interactions, lacking broader environmental context and diverse auditory signals. Furthermore, existing benchmarks primarily focus on visual cues, failing to assess integrated audio-visual reasoning. Even MLLMs capable of handling both modalities (e.g., Qwen2.5-Omni, Video-LLaVA2) are predominantly trained on exocentric data, leading to a significant gap in their ability to generalize to egocentric dynamics, often resulting in a bias toward visual signals and a neglect of audio cues.

Methodology: The EgoAVU Pipeline

To address these limitations, the authors introduce EgoAVU, a fully automated, scalable data engine designed to generate diverse, high-quality audio-visual-language data from public egocentric datasets (specifically Ego4D). The pipeline consists of four key components:

1. Egocentric Narration Enhancement:
   Instead of relying on a single MLLM to process audio and video simultaneously (which the authors found leads to modality bias and hallucination), EgoAVU employs a modular approach using specialized open-source MLLMs.

   • Visual: An image captioner (e.g., Qwen2.5-VL) processes the center frame to generate detailed spatial descriptions.
   • Temporal/Action: A video captioner (e.g., Qwen2.5-Omni without audio) generates coherent video-level narrations.
   • Audio: The same model acts as an audio captioner (without video) to produce detailed auditory narrations, capturing both foreground sounds (e.g., impacts) and background sounds (e.g., wind).
   • These uni-modal outputs are aligned temporally to enrich the original human narrations.

2. Video Filtering for Diversity:
   To ensure the dataset captures rich audio-visual dynamics, the system computes the Moving-Average Type–Token Ratio (MATTR) on the tokenized enhanced narrations. Videos with low MATTR scores (indicating repetitive or static descriptions) are filtered out, retaining only clips with high lexical diversity in objects, actions, and sounds.

3. Audio-Visual Narration Generation via Multimodal Context Graphs (MCG):
   To fuse the independent uni-modal narrations into a coherent joint description, EgoAVU constructs a Multimodal Context Graph (MCG). An LLM (LLaMA-70B) parses the enhanced narrations to extract structured relationships:

   • Interacted Objects: Objects physically manipulated by the user.
   • Background Objects: Visible but uninteracted items.
   • Foreground Sounds: Sounds correlated with specific actions or visible sources.
   • Background Sounds: Sounds without visual grounding.
     The LLM then uses the MCG and enhanced narrations to generate a unified, dense audio-visual narration that explicitly links actions, objects, and sounds.

4. QA Generation:
   Using the unified narrations, the system synthesizes question-answer pairs covering five categories:

   • Open-Ended: Source-Sound Association (SSA), Audio-Visual Segment Narration (AVSN), and Audio-Visual Dense Narration (AVDN).
   • Closed-Ended: Temporal Reasoning (TR) and Audio-Visual Hallucination (AVH).

Key Contributions

• EgoAVU-Instruct: A large-scale training dataset comprising 3 million audio-visual-language samples derived from 9,000 egocentric videos.
• EgoAVU-Bench: A manually verified evaluation benchmark containing 3,000 QA pairs across 900 distinct videos, covering diverse tasks and scenarios (cooking, construction, etc.) with an average video duration of 4 minutes.
• Automated Data Engine: A scalable pipeline that utilizes open-source models to generate joint audio-visual data without relying on closed-source APIs, ensuring reproducibility.
• Taxonomy of Tasks: A structured framework for evaluating egocentric audio-visual understanding, distinguishing between grounding, temporal reasoning, and hallucination.

Results

The authors evaluated existing MLLMs and their own fine-tuned models on EgoAVU-Bench:

• Baseline Limitations: Existing MLLMs (including Qwen2.5-Omni, VideoLLaMA2, and MiniCPM-o) demonstrated significant limitations. They exhibited a strong bias toward visual signals, often neglecting audio cues or failing to correctly associate sounds with their visual sources.
  • Source-Sound Association (SSA): Baseline models scored below 1.6/5 on LLM-as-a-judge evaluations.
  • Temporal Reasoning (TR): The best baseline achieved only 53.2% accuracy.
  • Hallucination (AVH): Models frequently hallucinated sound sources, with accuracy below 43%.
• Performance Gains: Fine-tuning MLLMs (specifically Qwen2.5-Omni 7B) on EgoAVU-Instruct resulted in substantial improvements:
  • Up to 113.3% relative performance improvement on open-ended tasks.
  • Up to 44.5% relative performance improvement on closed-ended tasks.
  • Significant reductions in hallucination rates across actions, objects, and sounds.
• Generalizability: The performance gains transferred to other egocentric benchmarks, such as EgoTempo (up to 28.1% accuracy gain) and EgoIllusion (up to 7.2% gain), while maintaining competitive performance on exocentric benchmarks like VideoMME and AVQA.

Significance and Claims

The paper claims that EgoAVU addresses a critical gap in embodied intelligence by providing the first large-scale, automated framework for generating and evaluating egocentric audio-visual understanding. The work demonstrates that:

1. Current MLLMs are visually biased: They struggle to integrate audio cues in dynamic, first-person contexts without specific training.
2. Modular data generation is effective: Using specialized models to extract uni-modal features before fusing them via context graphs yields higher quality joint data than direct multi-modal prompting.
3. Self-learning potential: The study highlights that leveraging the uni-modal capabilities of MLLMs can effectively improve their joint-modal capabilities, suggesting a path for future self-improving systems.

The authors acknowledge that their training data, while filtered, still contains noise inherent to open-source MLLM outputs, but posit that this will diminish as the underlying uni-modal models improve. The code, data, and benchmarks are released to the community to facilitate further research in this domain.