Technical Summary: Harnessing Streaming Video in the Wild

1. Problem Formulation

Current Vision-Language Models (VLMs) have achieved significant success in offline video understanding tasks (e.g., video question answering, temporal grounding) where the model processes a complete, pre-recorded clip to generate a one-shot response. However, a growing class of real-world applications—such as video-call assistants, live commentary, online surveillance, and embodied robots—operates on unbounded video streams.

In these "in-the-wild" scenarios, the fundamental assumptions of offline processing are violated:

• Causal Constraint: The model must respond based solely on previously observed frames, without access to future content.
• Unbounded Length: Streams can last for hours, making it impossible to retain all visual tokens in memory.
• Latency Constraints: Responses must be generated in real-time (sub-second latency) to remain synchronized with incoming frames.
• Proactive Interaction: The system must decide when to speak and when to remain silent, rather than generating output for every frame or waiting for a user query.

Existing methods typically address only one or two of these requirements (e.g., memory compression or latency optimization) in isolation, lacking a unified framework that simultaneously supports proactive interaction, long-horizon memory, and real-time inference. Furthermore, existing benchmarks often reduce evaluation to single-turn, offline-style queries, failing to assess core streaming behaviors like recalling distant evidence or maintaining coherence across turns.

2. Methodology

The authors propose a unified stack consisting of three components: a training dataset/objective, an inference framework (Harness), and an evaluation benchmark.

2.1 Streaming-Native VLM Training (Backbone)

To adapt VLMs to streaming interaction, the authors introduce Streaming-Train-248K, a high-quality dataset with per-second frame-text alignment.

• Data Construction: The dataset aggregates 248K samples from diverse sources (e.g., EpicKitchens, Ego4D, LiveWhisperX), covering daily living, work, and outdoor scenarios. Each one-second segment is paired with either an explicit response or a silence token, tagged with timestamps.
• Task Taxonomy: Tasks are categorized into Narration (emitting descriptions only for salient events) and Question Answering (categorized by answer invariance/variant and evidence timing: backward, present, forward).
• Training Objective: To address the severe class imbalance where silence tokens vastly outnumber response tokens, a weighted loss function is employed. Silence continuations are down-weighted ($w_{repeated\_silence} < 1$), while the transition from silence to response is up-weighted ($w_{response} > 1$). This encourages the model to learn when to break silence effectively.
• Special Tokens: The model is trained to emit two control tokens: </response> (to answer or narrate) and </silence> (to remain silent), enabling per-second proactive decision-making.

2.2 StreamingHarness (Deployment Infrastructure)

StreamingHarness is a plug-and-play inference framework designed to equip any VLM with streaming capabilities. It addresses memory and latency constraints through three mechanisms:

1. Three-Tier Memory Management:
   • Short-term: Retains raw vision tokens for the most recent $T_s$ seconds (e.g., 200s).
   • Mid-term: Compresses past short-term chunks into textual summaries ($T_m$ seconds).
   • Long-term: Aggressively consolidates mid-term summaries into blocks ($T_l$ seconds), capped at a fixed number of blocks with FIFO eviction.
   • Result: Supports up to 12 hours of context without unbounded memory growth.
2. Prefix-Aware KV Cache:
   • Designed for compatibility with efficient inference engines like vLLM.
   • The system performs a one-time prefill of the "Memory Text" (compressed history) at the start of each chunk.
   • Subsequent steps reuse the cached Key-Value (KV) states for the memory and prior turns, computing only the new frame and response. This enables sub-second latency even over long contexts.
3. Event-Driven Response Triggering:
   • The harness invokes the backbone every second.
   • The model judges whether the current evidence (current frame + context) warrants a response. If no new informative content is available, it emits </silence>; otherwise, it generates a grounded response.

2.3 Streaming-Eval (Benchmark)

To measure real-streaming capabilities, the authors introduce Streaming-Eval, a benchmark covering 138 videos across 15 categories (e.g., Sports, Cooking, News).

• Evaluation Axes:
  • Streaming Interaction: Assesses Proactive (when to speak), Punctual (timing), and Contextual (coherence across turns) behaviors.
  • Streaming Understanding: Assesses reasoning over Backward (past), Present, and Forward (future relative to query) evidence.
• Metrics:
  • Narration: Evaluated via win-rate against human/LLM judges comparing pacing, tone, and consistency with ground-truth ASR.
  • Question Answering: Uses SW-F1 (Streaming Weighted F1), a metric that jointly penalizes incorrect answers and mistimed responses (emitting answers when silence is required or missing the answer window). It assigns higher weights to True Positives and False Negatives to prioritize correctness while discouraging "always-respond" behaviors.

3. Key Contributions

1. Streaming-Train-248K: A large-scale dataset with per-second alignment and a tailored training objective to create Streaming-Native VLMs capable of proactive interaction.
2. StreamingHarness: A plug-and-play system providing 12-hour memory retention, sub-second latency, and proactive interaction (per-second decision making) compatible with vLLM.
3. Streaming-Eval: A comprehensive benchmark and metric suite (SW-F1) designed to evaluate models on real-world streaming tasks, moving beyond single-turn offline paradigms.
4. State-of-the-Art Performance: The proposed stack (8B Streaming-Native VLM + StreamingHarness) achieves superior results compared to strong closed-source models (e.g., Claude Opus 4.6, GPT 5.4, Gemini 3.1 Pro) in both narration and QA tasks.

4. Experimental Results

• Narration: The 8B streaming-native model equipped with StreamingHarness achieved an average win rate of 61.4% against four closed-source baselines. It significantly outperformed baselines in producing natural, live-broadcast-style commentary, avoiding the "over-narration" and hallucination common in offline-adapted models.
• Question Answering: On the Streaming-Eval QA tasks, the proposed system achieved substantial gains in Backward (retrieving distant evidence) and Forward (deferring response until evidence appears) tasks. The SW-F1 scores were significantly higher than all baselines, demonstrating the effectiveness of the memory system and proactive training.
• Efficiency: In a 2-hour sports broadcast test, the full StreamingHarness maintained Time-to-First-Token (TTFT) and End-to-End latency consistently below 1 second (the real-time threshold). In contrast, sliding window approaches stabilized at ~4 seconds, and full attention methods exceeded context limits within minutes. The prefix-aware KV cache was critical for reducing latency from ~4s to <1s.
• Ablation:
  • Adjusting the silence repetition weight ($w_{repeated\_silence}$) showed that a value of 0.8 provided the best balance, preventing the model from over-committing to silence while avoiding premature responses.
  • Adding StreamingHarness to offline models significantly improved their performance on Backward tasks, confirming the framework's value as a general-purpose enhancement for long-horizon understanding.

5. Significance

The paper argues that the transition from offline video understanding to deployable streaming intelligence requires a holistic approach addressing model capability, infrastructure, and evaluation simultaneously. By open-sourcing the data, harness, and benchmark, the authors aim to accelerate community progress in building VLMs that can operate effectively in unbounded, real-time environments. The work demonstrates that with the right training objectives and inference architecture, even smaller (8B) models can outperform massive closed-source systems in streaming scenarios while maintaining strict latency constraints.