Technical Summary: Semantic-Aware Adaptive Visual Memory for Streaming Video Understanding

Problem Definition

The paper addresses Online Streaming Video Question Answering (StreamingVQA), a task where a model must process a continuous, unbounded video stream and answer natural language queries in real-time as they arrive. Unlike offline settings where the full video is available, streaming imposes three critical constraints:

1. No Future Access: The model can only access frames up to the current timestamp $T$.
2. Query-Agnostic Encoding: Memory construction must occur without knowledge of the specific query, as queries arrive unpredictably.
3. Bounded Memory: The total memory footprint must remain constant regardless of video length.

Existing methods struggle with these constraints. Most rely on visual similarity heuristics for token compression, which fails to capture semantic importance, or they add retrieval mechanisms at the KV-cache level after compression is finalized. This separation makes it difficult to coordinate retention and retrieval, often leading to the loss of semantically critical but visually redundant information.

Methodology: SAVEMem

The authors propose SAVEMem, a training-free, dual-stage framework that decouples query-agnostic memory generation from query-aware retrieval. The system operates under a constant memory budget $B$ and utilizes a fixed "pseudo-question bank" to inject semantic priors without requiring model fine-tuning.

Stage 1: Semantic Memory Generation (Query-Agnostic)

In this stage, the model builds a three-tier streaming memory ($M_{short}$, $M_{mid}$, $M_{long}$) online as frames arrive.

• Semantic Prior: Instead of relying solely on visual similarity, SAVEMem scores each visual token $v$ against a fixed bank of pseudo-questions $Q$ (covering objects, actions, scenes, etc.) using a MaxSim late-interaction mechanism: $s(v) = \max_{q \in Q} \cos(v, q)$. This provides a query-agnostic estimate of semantic salience.
• Three-Tier Hierarchy with Selective Forgetting:
  • Short-term ($M_{short}$): A FIFO buffer preserving the most recent frames in full fidelity to anchor the current context.
  • Mid-term ($M_{mid}$): When $M_{short}$ fills, the oldest frame undergoes Temporal Semantic Pruning. Tokens are removed if they are well-represented by temporal neighbors, unless they have high semantic scores ($s(v)$) or are at scene boundaries.
  • Long-term ($M_{long}$): When $M_{mid}$ overflows, frames move here via Spatial Semantic Selection. This treats spatial coverage as a hard constraint, ranking tokens by their semantic scores to retain a dispersed but semantically ordered subset.
  • Global Budget Enforcement: A Selective Forgetting step ensures the total token count never exceeds $B$. If exceeded, the lowest-scoring tokens in $M_{long}$ are evicted.

Stage 2: Adaptive Memory Retrieval (Query-Aware)

Upon receiving a user query $q$ at time $T$, the system retrieves relevant frames from the frozen memory.

• Anchor-Conditioned Recency Gate: The system first checks if the query targets the immediate past. It computes the affinity between $q$ and the short-term anchor ($M_{short}$). If this affinity is high (relative to a running average), the system returns $M_{short}$ directly, bypassing the older tiers. This gate adapts per query based on the video's specific affinity distribution.
• Query-Aware Late Interaction: If the query requires historical context, the system computes a ColBERT-style late-interaction score between the query and tokens in $M_{mid} \cup M_{long}$. The score for a frame is the mean of the maximum cosine similarities between its tokens and the query tokens.
• Adaptive Retrieval: Frames are ranked by this score. The number of retrieved frames ($K$) adapts to the score dispersion: tightly clustered scores (diffuse evidence) result in retrieving more frames, while well-separated scores trigger aggressive filtering. The final set always includes $M_{short}$ plus the top-ranked historical frames.

Key Contributions

1. Dual-Stage Framework: A training-free architecture that separates semantic-aware compression from query-aware retrieval, allowing the two stages to be coordinated via a shared scoring mechanism (MaxSim) without joint training.
2. Semantic-Aware Compression: The introduction of a pseudo-question bank to guide long-term retention based on semantic salience rather than just visual similarity, ensuring that informative tokens are preserved even if they are visually redundant.
3. Adaptive Retrieval Scope: An anchor-conditioned recency gate that dynamically adjusts the retrieval scope (short-term vs. long-term) based on whether the query targets the present or the distant past.
4. Efficiency: The method achieves sub-linear token growth and nearly flat peak GPU memory usage relative to video length, transforming long-video processing into a budget-bounded problem.

Experimental Results

SAVEMem was evaluated on OVO-Bench, StreamingBench, and ODV-Bench using the Qwen2.5-VL backbone (3B and 7B parameters) without any training.

• OVO-Bench:
  • On the Qwen2.5-VL-7B backbone, SAVEMem improved the overall score from 52.27 to 62.69, outperforming recent training-free methods (e.g., FluxMem, HERMES) and competitive training-based approaches (e.g., ViSpeak).
  • Significant gains were observed in Real-Time Visual Perception (+10.41 points) and Backward Tracing (+10.41 points).
  • The method showed particular strength in tasks requiring fine-grained visual details (OCR, STU, ACR) and hallucination detection.
• StreamingBench & ODV-Bench:
  • Consistent improvements were observed across both backbones. On the 7B model, the ODV-Bench Static score improved by 8.7 points (48.3 to 57.0).
• Efficiency:
  • At 128 frames, SAVEMem reduced peak GPU memory usage by 48% compared to the backbone (18.5 GB vs. 35.8 GB), while maintaining sub-linear token growth.

Significance and Claims

The paper claims that decoupling query-agnostic semantic compression from query-aware adaptive retrieval is a practical direction for scaling streaming video understanding without the need for training. By injecting semantic signals into the compression phase and adapting the retrieval scope per query, SAVEMem addresses the "perception-memory trade-off" inherent in streaming. The authors emphasize that their approach allows models to handle unbounded streams on commodity GPUs by enforcing a constant memory budget while retaining semantically critical information.

The authors acknowledge limitations, noting that the method is inherently sensitive to the trade-off between selective retrieval and full coverage for tasks requiring exhaustive temporal scanning (e.g., counting). They also note that the pseudo-question bank is currently generic and not domain-adapted, and that the training-free design precludes joint optimization of the two stages.