Technical Summary: AVI-Bench: Toward Human-like Audio-Visual Intelligence of Omni-MLLMs

1. Problem Statement

While recent advances in Omni-Multimodal Large Language Models (Omni-MLLMs) have enabled the integration of text, vision, and audio, their Audio-Visual Intelligence (AVI) remains insufficiently evaluated. Existing benchmarks are often modality-specific (e.g., vision-language or audio-language) or lack a unified, structured framework to assess the multifaceted nature of real-world cross-modal scenarios. Current evaluations frequently rely on aggregate metrics that mask specific failure modes, such as modality imbalances, cognitive-stage bottlenecks, and poor adaptation to unfamiliar domains. Consequently, there is a lack of systematic tools to diagnose whether these models possess genuine human-like capabilities in perceiving, integrating, and reasoning across audio and visual inputs.

2. Methodology

2.1. AVI-Bench Framework

The authors introduce AVI-Bench, a cognitively inspired benchmark designed to evaluate Omni-MLLMs across three integrated cognitive stages and a unique extension for robustness testing:

1. Perception: Evaluates the detection and recognition of fundamental semantic entities (e.g., Audio/Visual Multi-instance Classification, Audio-Visual Localization, Audio-Visual Matching).
2. Understanding: Assesses the integration and reasoning over multimodal context (e.g., Audio-Visual Captioning, Cross-modal Retrieval).
3. Reasoning: Probes higher-order inference, including answering questions based on holistic understanding and grounding language references to specific objects/events (e.g., AVQA, Audio-Visual Language Grounding).
4. Primitive Sensation (AVI-Bench-PriSe): An extension evaluating model adaptation to unfamiliar, low-semantic stimuli. This stage tests whether models can perform basic sensory tasks (detecting variations in color, volume, texture, geometry) when semantic context is minimal, probing "primitive sensation" beyond standard training distributions.

The benchmark comprises 5,864 samples across 14 diverse tasks and 9 metrics, ensuring a balance between audio-dominant, visual-dominant, and collaboration-intensive tasks at each stage.

2.2. Four-Level Taxonomy

To move beyond simple aggregate scores, the paper proposes a four-level principled taxonomy for classifying AVI:

• Level 1 (Task-Adaptive): Baseline performance across all tasks.
• Level 2 (Modality-Adaptive): Measures the balance between audio and visual capabilities, penalizing models that act as "visual specialists."
• Level 3 (Stage-Adaptive): Evaluates whether reasoning is grounded in perceptual and understanding prerequisites, identifying bottlenecks where reasoning outpaces lower-level cognitive stages.
• Level 4 (Domain-Adaptive): Assesses robustness by comparing performance on familiar domains versus the unfamiliar domains of AVI-Bench-PriSe, using a harmonic mean to ensure neither domain dominates the score.

2.3. Evaluation Protocol

The study evaluates 28 Omni-MLLMs, including both closed-source (e.g., GPT-4o, Gemini series) and open-source models (e.g., Qwen2.5-Omni, Baichuan-Omni-1.5). The evaluation employs standardized prompts, fixed random seeds, and a double-confirmation mechanism for primitive sensation tasks to minimize hallucinations and ensure evaluation accuracy.

3. Key Contributions

• AVI-Bench & AVI-Bench-PriSe: A comprehensive, cognitively grounded benchmark spanning perception, understanding, reasoning, and primitive sensation, featuring 14 tasks and 9 metrics.
• Four-Level Taxonomy: A structured framework (Task-, Modality-, Stage-, and Domain-Adaptive) that provides a nuanced, interpretable view of model capabilities, revealing hidden limitations in aggregate scores.
• Empirical Insights: Extensive evaluation of 28 models revealing critical bottlenecks, including modality imbalances (visual dominance), stage-specific constraints (perception limiting reasoning), and significant gaps in unfamiliar domain adaptation.
• Human Performance Baseline: A pilot study comparing model performance against human subjects, highlighting substantial gaps, particularly in primitive sensation and grounding tasks.

4. Key Results and Observations

• Synergy and Bottlenecks: Performance across cognitive stages is positively correlated; however, perception and understanding act as critical bottlenecks. Models with strong reasoning but weak perception (or vice versa) fail to achieve robust AVI.
• Modality Imbalance: Most Omni-MLLMs excel at visual-dominant tasks but lag significantly in audio-dominant tasks, indicating a "visual specialist" bias rather than true audio-visual integration.
• Grounding Challenges: Fine-grained audio-visual grounding (e.g., localizing sound sources or referring expressions) remains a major challenge. Even top-performing models like Gemini-2.5-Pro achieve low scores (e.g., ~35-39%) on grounding tasks, with open-source models performing even worse.
• Primitive Sensation Gap: Models struggle significantly with low-semantic, unfamiliar stimuli. The performance gap between reasoning tasks and primitive sensation tasks is substantial (ranging from 47.6% to 82.7%), suggesting models rely heavily on semantic priors rather than genuine sensory adaptation.
• Scale and Architecture: Larger models generally outperform smaller ones, but architectural choices (e.g., mixture-of-LoRA) and training strategies significantly impact performance, sometimes allowing smaller models to outperform larger counterparts in specific dimensions.

5. Significance and Claims

The paper claims that AVI-Bench provides a principled evaluation framework to guide the development of more robust and generalizable Omni-MLLMs. By shifting from fragmented task-specific metrics to a holistic, cognitively aligned assessment, the benchmark:

• Reveals critical bottlenecks (modality imbalance, weak grounding, poor domain adaptation) that hinder progress toward Artificial General Intelligence (AGI).
• Offers a structured taxonomy that enables fine-grained diagnosis of failure modes, distinguishing between models that merely fit patterns and those exhibiting human-like cognitive integration.
• Establishes a rigorous foundation for future research, emphasizing the need for balanced audio-visual intelligence and robustness beyond common training distributions.

The authors position AVI-Bench not as a tool for training, but as a diagnostic resource to support safety research, model auditing, and the responsible advancement of multimodal AI systems.