Technical Summary: OmniAgent for Omnimodal Audio-Video Understanding

1. Problem Statement

Omnimodal Large Language Models (OmniLLMs) have achieved significant progress by unifying visual and audio encoders. However, they face critical limitations in fine-grained cross-modal understanding and multimodal alignment. Existing end-to-end models often struggle to allocate attentional resources adaptively between modalities; they are constrained by fixed token budgets and joint optimization objectives that prevent dynamic prioritization of specific modalities or adjustment of processing granularity. Consequently, they frequently fail to resolve queries requiring precise temporal grounding or high-resolution visual inspection.

Furthermore, previous agent-based approaches for video understanding rely on rigid, static workflows or dense frame-captioning. These methods incur substantial computational overhead, generate irrelevant captions, and lack the flexibility to dynamically plan attention based on the specific requirements of a user query. While audio naturally provides accurate temporal grounding for salient events, current systems fail to leverage this efficiently to guide visual analysis.

2. Methodology: OmniAgent

The authors introduce OmniAgent, the first fully active perception agent designed for omnimodal audio-video understanding. Unlike passive models, OmniAgent treats strong single-modal models as callable tools and employs a recursive "Think-Act-Observe-Reflect" loop to dynamically orchestrate multimodal reasoning.

Core Architecture

The system operates as a sequential decision-making process where an LLM (the "brain") autonomously determines the optimal modality and granularity for each step. The process involves:

1. Active Thinking: The agent assesses the cross-modal dependency of a query to decide whether to "listen" (audio) or "watch" (video) and selects the appropriate tool.
2. Action & Observation: The agent executes a selected tool on the corresponding modality stream, receiving an observation (e.g., a transcript, a visual description, or a timestamp).
3. Reflection: The agent synthesizes current observations with historical context to perform a cross-modal consistency check. If evidence is insufficient or discrepancies exist, the agent re-initiates the thinking cycle to gather more data. This loop continues until a final answer is synthesized.

Modality-Aware Expert Toolset

OmniAgent utilizes a comprehensive library of tools stratified by modality and granularity:

• Video Perception Tools:
  • Global QA (TVGA): Uses sparse frame sampling for coarse temporal localization and global context.
  • Clip QA (TVCA): Analyzes specific temporal windows at higher sampling rates and resolutions for fine-grained visual reasoning.
• Audio Perception Tools:
  • ASR (TASR): Transcribes dialogue with precise timestamp alignment.
  • Global Caption (TAGC): Summarizes the acoustic environment.
  • Audio QA (TAQ): Extracts specific acoustic details via targeted inquiries.
• Event Perception Tools (The Novel Contribution):
  • Event List (TEL) & Event Location (TELO): A novel audio-guided event localization strategy. The agent processes the entire audio stream to extract a discrete list of sound events, establishing temporal anchors. This allows the agent to pinpoint the exact time a salient event occurs (e.g., a specific sound or phrase) and subsequently invoke high-resolution video tools on that specific window.

Coarse-to-Fine Reasoning Paradigm

The agent employs a coarse-to-fine audio-guided perception paradigm. It typically begins with low-cost audio cues (ASR or Event Location) to localize the relevant temporal segment. Once the time window is identified, the agent switches to high-cost, high-resolution visual tools (Clip QA) to verify details. This approach mimics human cognition, avoiding the computational cost of processing the entire video at high fidelity.

3. Key Contributions

1. OmniAgent Framework: The first active perception agent for omnimodal understanding that dynamically modulates attention between auditory and visual modalities via a self-reflective mechanism, effectively solving cross-modal alignment issues without rigid training.
2. Audio-Guided Event Localization: The introduction of a novel algorithm that leverages audio cues to temporally localize events, serving as a precise proposal mechanism for subsequent fine-grained visual analysis.
3. Comprehensive Toolset: The construction of a modality-specific toolkit (Video, Audio, Event) that allows the agent to adaptively select spatial and temporal focus ranges.

4. Experimental Results

The authors evaluated OmniAgent on three benchmarks: Daily-Omni (short-form), OmniVideoBench (long-form), and WorldSense (medium-length, multi-domain).

• Performance: OmniAgent achieved State-of-the-Art (SoTA) performance across all benchmarks without training.
  • On Daily-Omni, it reached 82.71% accuracy, surpassing the strong closed-source baseline Gemini 2.5-Flash (72.70%) and the open-source Qwen3-Omni-30B (72.08%) by margins of 10–20%.
  • On OmniVideoBench, it achieved 59.10% accuracy, significantly outperforming Qwen3-Omni-30B (38.40%) and other end-to-end models.
  • On WorldSense, it achieved 61.2% average accuracy, leading across various domains including Tech, Culture, and Daily Life.
• Ablation Studies:
  • Tool Necessity: Removing Video Clip QA or Event Location tools caused significant performance drops, confirming the necessity of fine-grained visual analysis and audio-guided localization.
  • Model Choice: The choice of the underlying LLM for the "brain" and the specific tool models (e.g., using Gemini-2.5 for event localization due to its superior temporal grounding) critically impacts performance.
  • Reasoning Behavior: Analysis showed that successful agents (like those using OpenAI o3) follow a strategic pattern: establishing global context first, then using event localization to narrow the scope, and finally performing fine-grained visual verification. Models that converged prematurely on coarse-grained evidence (e.g., GPT-4o) performed poorly.

5. Significance and Claims

The paper claims that OmniAgent represents a paradigm shift from passive response generation to active multimodal inquiry. By explicitly decoupling modalities into callable tools, the system circumvents the alignment bottlenecks inherent in end-to-end models.

The authors emphasize that their approach:

• Solves Cross-Modal Alignment: By using audio to guide visual attention, the system avoids the need for complex joint alignment training.
• Enables Fine-Grained Understanding: The ability to dynamically increase spatial and temporal resolution only where necessary allows for detailed reasoning that static models miss.
• Offers Transparency: The decision-making process (tool selection, reflection) is transparent and explainable, unlike the "black box" nature of end-to-end fusion.

The work is presented as a foundational step toward omnimodal agentic algorithms, suggesting that future research should focus on training native omnimodal agents capable of self-calling tools to further reduce inference costs and improve efficiency. The authors strictly limit their evaluation to open-source benchmark datasets to address privacy concerns associated with real-world video analysis.