Technical Summary: StreamOV

Problem Statement

The paper addresses the emerging challenge of streaming omni-video understanding, a setting where models must process synchronized audio-visual signals in real-time as they arrive, rather than accessing a complete video offline. Current state-of-the-art omni-modal models are designed for offline inference, creating two fundamental mismatches in streaming scenarios:

1. Memory Management: Existing methods lack robust mechanisms to manage continuously growing audio-visual contexts over long horizons without exceeding computational budgets.
2. Proactive Interaction: Offline models cannot autonomously determine when to respond. They either generate responses for every query (ignoring the need for silence) or rely on inefficient mechanisms like explicit <silence> tokens (which disrupt pre-trained generation distributions) or external routers (which lack deep multimodal reasoning capabilities).

Furthermore, the field lacks a comprehensive benchmark capable of evaluating continuous, multi-turn, online audio-visual interactions that include proactive response triggering and intentional silence.

Methodology: StreamOV Framework

The authors propose StreamOV, a framework built upon an Omni-modal Large Language Model (MLLM) designed for efficient online reasoning with bounded memory and proactive response triggering.

1. Multimodal Evidence-Guided Memory

To handle the infinite stream under a fixed memory budget, StreamOV introduces a Long-Short Term Memory update mechanism driven by a dual-perspective evidence construction strategy:

• Query-Agnostic Dynamics: Captures intrinsic stream dynamics using visual change ($S_v$), audio saliency ($S_a$), and audio-visual co-bursts ($S_{cob}$).
• Query-Aware Semantics: Measures the relevance between the current query and the stream using pre-trained encoders for visual ($S_{qv}$) and audio ($S_{qa}$) semantic scores. A gating mechanism refines audio relevance to prioritize segments with physically salient audio or synchronized events.
• Evidence Routing: The system constructs three disentangled evidence types: visual-only, audio-only, and audio-visual-aligned.
• Memory Update:
  • Short-term Memory ($M^S_t$): Retains dense observations from the current temporal window.
  • Long-term Memory ($M^L_t$): Maintains a sparse, fixed-budget set of informative observations selected from the entire history based on evidence scores.
  • The final memory state is the union of these buffers, serialized chronologically and fed into the frozen MLLM.

2. Hidden-State-Driven Response Trigger

Instead of relying on external routers or generating non-semantic <silence> tokens, StreamOV employs an MLLM-as-a-trigger mechanism:

• Mechanism: The model probes the hidden states of the early decoding steps (specifically the state after prefilling, $h_{t,0}$) to predict whether to Respond or Wait.
• Implementation: A lightweight Cross-Attention Trigger module aggregates these hidden states to output binary triggering logits.
• Advantage: This approach leverages the MLLM's intrinsic cognitive readiness encoded in its hidden states, avoiding the disruption of pre-trained distributions and the reasoning limitations of small auxiliary models.

Key Contributions

1. SOVBench: The first comprehensive benchmark for online, multi-turn omni-modal evaluation. It consists of two components:
   • SOVBench-O: Evaluates continuous audio-visual comprehension across Real-Time, Recall, and Proactive interaction paradigms (including cognitive proactivity where the model must infer intent).
   • SOVBench-T: A binary classification benchmark specifically for Response Triggering, testing a model's ability to respond when evidence is present and remain silent when it is absent.
2. StreamOV Framework: A novel architecture integrating multimodal evidence routing, long-short term memory, and a hidden-state-driven trigger to enable efficient streaming reasoning.
3. Empirical Validation: Extensive experiments demonstrating that StreamOV achieves state-of-the-art performance across streaming, offline, and visual-only benchmarks.

Experimental Results

The authors evaluated StreamOV (based on Qwen3-Omni-30B-A3B with a frozen backbone and a 18.9M parameter trigger) against various baselines, including offline Qwen models, online streaming models (e.g., ROMA, Flash-VStream), and closed-source models (Gemini).

• SOVBench Performance: StreamOV achieved 81.6% average accuracy on SOVBench-O (Audio-Visual Context) and 83.8% on SOVBench-O (Audio-Visual Context & QA context), outperforming the strong offline baseline Qwen3-Omni-30B (73.7% and 79.9% respectively).
• Proactive Triggering: On SOVBench-T, StreamOV achieved an F1 score of 90.5, significantly outperforming the Qwen3-Omni baseline with a trigger (81.4) and other online models.
• StreamingBench: StreamOV improved performance by +7.6% over Qwen3-Omni and +22.5% over ROMA on the Audio-Visual QA subset.
• Offline Benchmarks: StreamOV demonstrated competitive or superior performance on offline benchmarks like Video-Holmes and Daily-Omni, indicating its effectiveness extends beyond purely streaming scenarios.
• Ablation Studies: Results confirmed that Query-Aware, Query-Agnostic, and Long-Memory components are all essential, with the Long-Memory module providing the most significant performance boost.

Significance and Claims

The paper claims that StreamOV bridges the gap between offline omni-modal understanding and the requirements of real-world streaming interactions. By introducing a mechanism for proactive response triggering without external routers or silence tokens, and by establishing SOVBench as a standard for evaluating continuous, multi-turn audio-visual reasoning, the work provides a foundational step toward models that can "listen and watch" continuously, deciding when to speak based on the sufficiency of evidence. The authors emphasize that their method achieves state-of-the-art results while maintaining bounded memory and computational efficiency, making it suitable for long-horizon streaming applications.