Technical Summary: SelectStream

Problem Statement

Streaming video understanding requires Vision-Language Models (VLMs) to answer queries at any moment during an ongoing video stream using only the observed prefix, subject to fixed memory, context, and computation budgets. While recent work has attempted to solve this by adding external memory banks, retrieval modules, or visual token compression, these methods often rely on the assumption that "richer memory" automatically leads to better understanding.

The authors argue that indiscriminate history injection can dilute current-scene perception, a phenomenon where relevant evidence is mixed with irrelevant content, weakening the model's ability to reason. Conversely, simple recent-window baselines (which discard older evidence) fail when critical information lies outside the immediate context. The core challenge is not whether to use memory, but how to selectively allocate it: deciding when to write new evidence, what to preserve when capacity is full, how to retrieve evidence for a specific query, and how much historical context to expose during generation without growing the context window with stream length.

Methodology: SelectStream

SelectStream is a selective latent-memory framework that formulates streaming memory as budgeted online latent evidence allocation. It operates on a frozen VLM backbone and manages history through a dynamic latent evidence graph ($G_t = (M_t, E_t)$) rather than replaying raw frames or converting history into text summaries. The system is governed by three coordinated mechanisms:

1. Surprise-driven Adaptive Windowing (SAW) – When to Write

SAW determines when to close a video segment and write it to memory. Instead of fixed intervals, it assigns coarse memory entries to stable intervals and finer resolution near surprising transitions.

• Mechanism: It calculates a surprise score ($s_t$) based on attention shifts (JS divergence of attention maps) and feature changes (cosine distance of projected embeddings).
• Triggers: A segment closes when it reaches a minimum length ($L_{min}$) and one of three conditions is met: a surprise spike ($\bar{s}_t > \theta_{high}$), accumulated surprise energy exceeds a budget ($B_s$), or the maximum segment length ($L_{max}$) is reached.

2. Latent Visual Memory (LVM) – What to Preserve

LVM manages a fixed-capacity memory graph where projected visual embeddings are stored as latent nodes.

• Writing: When a segment is closed, its embedding is compared to active nodes. If semantically similar and low in surprise, it updates an existing node via a gated update mechanism (preventing direct replacement of useful history). Otherwise, a new node is created.
• Consolidation: When the memory budget ($N$) is exceeded, the system performs priority-preserving consolidation. It merges the pair of nodes with the smallest penalty, which balances semantic redundancy (similarity) against priority factors (surprise, access frequency, and recency). This ensures that surprising or frequently accessed evidence is preserved.

3. Query-conditioned Graph Attention Reasoning (GAR) – How to Read

When a query arrives, the model retrieves a compact subgraph rather than reasoning over the entire history.

• Routing: The query is encoded and used to score active nodes. A subgraph is routed from top-$k$ seeds through temporal and semantic edges, constrained by a subgraph budget ($B$).
• Reasoning: A Graph Attention Network (GAR) refines the node states by aggregating temporal and semantic context from neighbors.
• Injection: The top-$M$ refined nodes are calibrated and injected as latent evidence tokens into the frozen VLM's decoder. This allows the VLM to access historical evidence without increasing the context length proportional to the stream duration.

Key Contributions

1. Formulation: The paper formulates selective remembering in streaming video as a budgeted online latent evidence allocation problem, explicitly addressing the trade-off between retaining current-scene perception and accessing long-range history.
2. Architecture (SelectStream): A dynamic latent-memory architecture that writes projected VLM embeddings into a fixed-capacity graph. It utilizes surprise-driven windowing for adaptive writing, priority-aware consolidation for preservation, and query-conditioned graph reasoning for retrieval.
3. Interface: A design that injects calibrated latent evidence tokens into a frozen VLM, avoiding raw frame replay or text summarization, thereby maintaining a bounded inference cost regardless of stream length.

Experimental Results

SelectStream was evaluated on streaming and offline video benchmarks using Qwen2.5-VL-7B and Qwen3-VL-8B backbones.

• Streaming Benchmarks:
  • StreamingBench: Achieved 82.67% accuracy (Qwen3-VL-8B), outperforming strong recent-window baselines (e.g., Qwen3-VL-8B + 4f at 80.59%) and prior streaming memory methods.
  • OVO-Bench: Achieved 67.03% overall accuracy. Notably, it improved Backward Tracing (BT) scores significantly (from 54.00% to 62.20% on Qwen3-VL-8B), demonstrating superior ability to utilize prior visual context while maintaining high Real-Time Visual Perception (82.76%).
• Offline Benchmarks:
  • On VideoMME, MLVU, and MVBench, SelectStream achieved an average accuracy of 74.4% (Qwen3-VL-8B), showing that the method preserves general video understanding capabilities and does not trade offline performance for streaming efficiency.
• Efficiency:
  • The model maintains nearly flat query latency (TTFT) and peak GPU memory as the number of processed frames increases, as the active memory state is bounded by $N$ and retrieval is bounded by $B$ and $M$.

Significance and Claims

The paper claims that SelectStream addresses the "evidence dilution" problem inherent in indiscriminate memory injection. By keeping the current observation directly visible to the frozen VLM and exposing historical information only through a compact, query-conditioned evidence budget, the model achieves a superior trade-off between long-range memory and current-scene perception.

The authors position SelectStream as a scalable direction for streaming video understanding under fixed resource constraints. The results suggest that selective latent memory provides benefits beyond simply retaining the latest frames, enabling robust history-dependent reasoning without the computational overhead of growing context windows. The framework is designed to be a general memory layer for frozen video-language backbones, separating online memory management from query-time generation.