Technical Summary: Agentic Active Omni-Modal Perception for Multi-Hop Audio-Visual Reasoning

Problem Statement

Current Omni-Modal Large Language Models (Omni-LLMs) face significant challenges in multi-hop audio-visual reasoning, particularly when relevant evidence is sparse, temporally dispersed, and distributed across both audio and visual streams. Existing benchmarks often fail to isolate this specific capability, typically involving limited modalities, short temporal segments, or single-modality decisive evidence. Consequently, it remains unclear whether Omni-LLMs can effectively integrate sparse, cross-modal evidence distributed across multiple temporal segments to answer complex questions. Furthermore, existing agentic approaches for active perception often rely on costly proprietary models or require extensive training, limiting their applicability to open-source Omni-LLMs under low-resource settings.

Methodology

1. MOV-Bench: A Benchmark for Multi-Hop Reasoning

To rigorously evaluate this capability, the authors introduce MOV-Bench (Multi-hop Omni Video Benchmark).

• Dataset Construction: Built upon the Fine-Video dataset, MOV-Bench contains 519 carefully curated multiple-choice questions.
• Design Principles:
  • Multi-hop Temporal Reasoning: Questions require evidence from 2 to 4 distinct temporal segments, preventing models from relying on single-moment perception.
  • Cross-modal Integration: Questions are constructed such that neither the audio nor visual stream alone is sufficient; models must jointly interpret visual events, speech content, and audio cues.
  • Reasoning Themes: Questions cover five distinct reasoning patterns: causal, referential, relational, hypothetical, and intent reasoning.
• Quality Control: A two-stage filtering process (language-only filtering to remove bias, followed by human verification) ensures questions require genuine cross-modal multi-hop reasoning.

2. AOP-Agent: Active Omni-Modal Perception Framework

To address the limitations of passive processing, the authors propose AOP-Agent, an efficient agentic framework designed for open-source Omni-LLMs without additional training or proprietary components.

• Hierarchical Omni-modal Memory: Instead of processing raw video streams directly, AOP-Agent organizes video content into a structured memory:
  • Fine-grained clips: Preserving local details.
  • Mid-level segments: Merged clips (≤30s) compressed at lower resolution.
  • Structured Metadata: For each segment, the system generates visual keypoints, audio keypoints, retrieval keywords, and segment-level descriptions.
  • Global Summary: A synthesized description of the entire video.
• Observe-Reflect-Replan Loop: The framework employs a multi-agent collaboration process:
  • Planner Agent ($A_P$): Determines the next observation target based on the current reasoning state, selecting specific video regions to inspect.
  • Observation Tools: A suite of five complementary tools allows the agent to navigate the hierarchical memory:
    • Description-based: Semantic matching for high-level events.
    • Keyword-guided: Lexical retrieval for explicit entities/actions.
    • Keypoint-guided: Matching visual/audio keypoints for cross-modal clues.
    • Neighbor Observation: Expanding to temporally adjacent segments.
    • Fine-grained Observation: Drilling down for local verification.
  • Reflector Agent ($A_R$): Evaluates whether current observations are sufficient. If not, it provides feedback to guide replanning; if yes, it triggers the final reasoning.
  • Reasoner Agent ($A_A$): Synthesizes accumulated evidence to generate the final answer.

Key Contributions

1. MOV-Bench: A new benchmark specifically targeting multi-hop audio-visual reasoning across temporally dispersed segments, revealing that current Omni-LLMs struggle to locate and integrate sparse cross-modal evidence.
2. AOP-Agent: A low-resource agentic framework that enables open-source Omni-LLMs to perform active perception. It achieves this through hierarchical memory construction and a collaborative observe-reflect-replan loop, eliminating the need for additional training or proprietary models.
3. Empirical Validation: Demonstrations that AOP-Agent consistently improves reasoning performance on both MOV-Bench and OmniVideoBench, with significant gains on long videos and reasoning-intensive questions.

Experimental Results

Experiments were conducted on MOV-Bench and OmniVideoBench using various open-source Omni-LLMs (e.g., Qwen3-Omni-Instruct, Qwen2.5-Omni-7B).

• Baseline Limitations: Direct inference on MOV-Bench revealed that current Omni-LLMs achieve relatively low accuracy (e.g., Qwen3-Omni-Instruct at ~52.8%). Performance degrades significantly as video length increases and as the number of reasoning hops grows (e.g., dropping to ~44% on 4-hop questions).
• AOP-Agent Performance:
  • General Improvement: AOP-Agent consistently outperformed direct inference baselines across different model backbones. For instance, on long videos with Qwen3-Omni-Instruct, accuracy improved from 45.50% to 60.85%.
  • Comparison with Agentic Frameworks: Unlike existing agentic frameworks (OmniAgent, ActiveVideoPerception) which performed worse than direct inference when adapted to open-source models, AOP-Agent achieved superior results. This suggests that existing methods, often designed for powerful proprietary models, amplify error accumulation in weaker open-source models without the specific structural support AOP-Agent provides.
  • Ablation Studies:
    • Components: Both the Planner and Reflector agents contributed to performance, with the full loop yielding the best results.
    • Model Assignment: Stronger planning and reflection capabilities were found to be more critical than the strength of the final reasoner, highlighting that effective evidence selection precedes successful reasoning.
    • Observation Rounds: Performance improved with increased observation rounds (up to 3) but saturated or slightly degraded with excessive rounds, indicating a trade-off between evidence recovery and noise accumulation.

Significance and Claims

The paper claims that the primary bottleneck for current Omni-LLMs in multi-hop reasoning is not merely the reasoning capability itself, but the ability to effectively locate, preserve, and integrate sparse, temporally dispersed evidence.

By shifting from passive video processing to active, iterative perception, AOP-Agent allows open-source models to selectively inspect question-relevant evidence. The authors posit that this approach offers a scalable and effective paradigm for long-video omni-modal reasoning, particularly in scenarios where decisive evidence is sparse. The work demonstrates that complex multi-hop reasoning can be achieved without costly proprietary models or large-scale training, provided the system is equipped with hierarchical memory and a structured active perception loop.

Limitations Acknowledged:

• The scale of MOV-Bench is limited by annotation costs.
• The multi-round loop introduces inference overhead.
• The system remains susceptible to hallucinations in the underlying memory base, which can propagate errors.
• The current memory construction is offline-based, limiting applicability to real-time streaming scenarios.