Technical Summary: StreamReady

Problem Statement

Multimodal Large Language Models (MLLMs) have advanced video understanding, particularly for short clips, but struggle with long videos due to difficulties in reasoning over extended temporal contexts. While recent efforts have extended MLLMs to longer videos, most operate in an offline setting where the full video is available during inference. In contrast, streaming video understanding is an online variant where frames arrive sequentially, and the model never sees the full video.

A critical gap exists in current streaming approaches: they primarily focus on answer correctness in past-dependent (causal) scenarios where evidence is already available. However, many real-world applications (e.g., surveillance, robotics, assistive systems) require proactive (future-dependent) reasoning, where a question is posed before the supporting visual evidence appears. In such settings, models must determine when to answer. Answering too early constitutes unsupported speculation (hallucination), while answering too late reduces real-time utility. Existing benchmarks and methods lack mechanisms to evaluate or enforce the timing of answers relative to the appearance of evidence.

Methodology: StreamReady Framework

The authors propose StreamReady, a framework designed to unify temporal reasoning with explicit answer timing through a "readiness-aware" formulation.

1. Readiness-Aware Formulation

The core contribution is a new evaluation metric called the Answer Readiness Score (ARS). Unlike standard accuracy, ARS jointly evaluates correctness and timing using asymmetric penalties:

• Early Penalty (EP): A harsh penalty for answering before the evidence window begins, discouraging speculation.
• Late Penalty (LP): A milder penalty for answering after the evidence window ends, tolerating slight delays.
• Effective Accuracy: Defined as $Acc_e = Acc \times ARS$, this metric rewards models that are both correct and timely.

2. Framework Architecture

StreamReady operates without requiring full-video access or heavy auxiliary models. It consists of three main components:

• Hierarchical Memory Storage:

  • Visual Memory Tree ($M_V$): A multi-level structure to manage long streams efficiently.
    • $M_{V1}$: A FIFO buffer of recent raw frame embeddings (short-term details).
    • $M_{V2}$: A mid-level summary of centroids derived from $M_{V1}$ via K-means clustering.
    • $M_{V3}$: A coarse prototype set abstracted from $M_{V2}$, updated via Exponential Moving Average (EMA) to handle stable scenes and drift.
  • Contextual Memory Bank ($M_C$): Stores embeddings of past questions and their corresponding answer representations to support multi-turn reasoning and semantic consistency.

• Query-Aware Reasoning:

  • Upon receiving a question, the model activates a dual-branch Q-Former.
  • Short-term Branch ($Q_s$): Attends to raw frames in $M_{V1}$ for immediate context.
  • Long-term Branch ($Q_\ell$): Performs coarse-to-fine retrieval. It first scores prototypes in $M_{V3}$ to identify relevant regions, then refines the search to specific centroids in $M_{V2}$.
  • Contextual Reasoning: A soft-gating mechanism retrieves relevant past QA pairs from $M_C$ and fuses them with visual features via cross-attention.

• Lightweight Readiness Mechanism:

  • A learnable <RDY> token is appended to the long-term reasoning representation ($z_\ell$).
  • A Readiness Head (a lightweight MLP) monitors this token, outputting a readiness score $R_{pred} \in [0, 1]$.
  • Inference: The model only triggers the LLM to generate an answer when $R_{pred}$ exceeds a threshold. Otherwise, it continues observing.
  • Training: Since ground-truth evidence timestamps are unavailable during training, the authors use weak pseudo-supervision. They define pseudo-positive regions (where the reasoning representation is highly similar to visual memory centroids) and pseudo-negative regions. A pairwise contrastive loss trains the readiness head to assign higher scores to positive regions, while a temporal coherence regularizer ensures smooth transitions.

Key Contributions

1. Formalization of Readiness-Aware Streaming: The paper introduces a formulation where the goal is not just to answer correctly, but to answer precisely when sufficient evidence appears, formalized via the Answer Readiness Score (ARS).
2. StreamReady Framework: A novel architecture integrating hierarchical visual memory, query-aware coarse-to-fine retrieval, and a lightweight readiness token that decides when to respond based on internal evidence sufficiency.
3. ProReady-QA Benchmark: A new benchmark specifically designed for proactive streaming scenarios. It features:
   • 5k proactive multi-turn QA pairs across local and global contexts.
   • Annotated answer evidence windows (start and end timestamps) to enable ARS evaluation.
   • Five task types: Sequential Steps Recognition, Repetitive Event Count, Clues Reveal Responding, Causal Trigger Detection, and Goal-State Detection.

Experimental Results

The authors evaluated StreamReady on ProReady-QA and eight other benchmarks (four streaming, four offline long-video).

• ProReady-QA Performance: StreamReady achieved the highest Effective Accuracy and ARS across all five tasks, outperforming the best baseline (StreamBridge) by approximately 3% in accuracy and 9% in ARS on average. It showed particularly strong gains in tasks requiring precise timing (REC, GSD, CTD).
• Streaming Benchmarks: StreamReady consistently outperformed prior online methods (e.g., Flash-VStream, StreamBridge) on proactive and non-proactive streaming benchmarks (StreamingBench, OVOBench), achieving up to 5% gains on proactive tasks.
• Offline Long-Video Benchmarks: Despite being designed for streaming, StreamReady generalized well to offline benchmarks (VideoMME, MLVU, MVBench, EgoSchema), outperforming existing offline models. This suggests its hierarchical memory and reasoning mechanisms are beneficial even when timing is not explicitly penalized.
• Efficiency: Analysis showed StreamReady maintains stable latency and memory usage as video length increases, unlike baseline models that suffer from quadratic attention growth or heavy token compression overhead.

Significance and Claims

The paper claims that StreamReady establishes a unified foundation for advancing answer timing in streaming video understanding. Its significance lies in:

• Shifting the paradigm from "what to answer" to "what and when to answer," addressing a critical gap in real-world streaming applications.
• Demonstrating that timing can be learned via a lightweight mechanism coupled with evidence tracking, without relying on non-deterministic auxiliary models or heavy compute.
• Providing ProReady-QA as the first systematic testbed for evaluating the temporal appropriateness of streaming models, moving beyond simple accuracy metrics.
• Showing that readiness-aware mechanisms improve not only proactive performance but also general video understanding capabilities across diverse benchmarks.

The authors conclude that their work moves the field toward models that are not only accurate but also evidence-grounded and timely, a prerequisite for effective deployment in real-time, long-horizon scenarios.