💻 computer science

Towards a Dynamic and Fixed-budget Memory Bank for Efficient Streaming Video Understanding

This paper proposes CausalMem, a training-free approach that constructs a dynamic, fixed-budget memory bank using online semantic basis updates to efficiently compress and preserve streaming video information, significantly outperforming existing methods in both streaming and offline video understanding tasks while achieving over 20x token compression.

Original authors: Baiyang Song, Yuli Lin, Qiong Wu, Tao Chen, Jun Peng, Xiao Chen, Yiyi Zhou, Rongrong Ji

Published 2026-06-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Baiyang Song, Yuli Lin, Qiong Wu, Tao Chen, Jun Peng, Xiao Chen, Yiyi Zhou, Rongrong Ji

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to watch a live, 24-hour news feed on your phone, but your phone's memory is tiny. Every time a new scene appears, your phone tries to save a picture of it. If you keep saving every single frame, your phone will run out of space in seconds, and it will get so slow it can't even answer your questions about what's happening.

This is the problem CausalMem solves for AI models that watch videos.

Here is how the paper explains it, using simple metaphors:

The Problem: The "Infinite Scroll" vs. The "Tiny Backpack"

Current AI models (called MLLMs) are great at understanding videos, but they have a major flaw when it comes to streaming (watching a video as it happens, frame by frame).

The Issue: As the video plays, the AI tries to remember everything. It's like trying to carry a backpack that gets heavier with every step you take. Eventually, the backpack is so heavy (too much data) that the AI collapses (runs out of memory) or moves too slowly to answer questions.
The Catch: You can't just look at the whole video first to decide what's important (like a human watching a movie and then picking the best scenes). In a live stream, you don't know what's coming next. You have to make decisions on the fly.

The Solution: The "Smart Librarian"

The authors created a new method called CausalMem. Think of it as a Smart Librarian who manages a very small, fixed-size bookshelf (the "Memory Bank").

The Fixed Shelf: No matter how long the video is (1 minute or 10 hours), the bookshelf only holds a specific number of books (a fixed budget of "tokens"). It never gets bigger.
The "Semantic Basis" (The Librarian's Mental Map): As new frames arrive, the AI doesn't just save them blindly. It builds a "mental map" of the main themes and shapes it has seen so far. This is called the Online Semantic Basis.
- Analogy: Imagine the librarian knows the general "vibe" of the story so far. If a new scene is just more of the same background (redundant), the librarian knows, "I already have this in my head; I don't need to write it down."
The "Residual" Check (What's New?): The system calculates the "residual" or the "leftover" information.
- Analogy: If the new scene is just a blue sky, and the librarian already has a picture of a blue sky, the "leftover" is tiny. That scene is redundant and gets thrown away.
- If the new scene shows a sudden explosion or a new character, the "leftover" is huge. That scene is informative and gets a spot on the shelf.
The "Recency" Rule: The system also remembers that recent events are important. Even if a scene isn't super unique, if it just happened, it stays on the shelf for a little while so the AI doesn't forget what's happening right now.

The Result: A Super-Efficient Memory

The paper claims that by using this "Smart Librarian" approach, they can:

Compress Data: They can watch a 1-hour video and store it using only 12,000 "tokens" (a tiny amount of data). This is a 20x compression compared to saving everything.
Save Space: The memory used is only about 82 MB (roughly the size of a few high-quality photos), which is tiny for an hour of video.
Work Faster: Because the AI isn't trying to process millions of frames, it answers questions much faster and uses less computer power.
Be Accurate: Even with this tiny memory, the AI understands the video better than other methods. It scored higher on tests for both live streaming and pre-recorded videos.

In a Nutshell

CausalMem is like a human brain watching a live stream. Instead of remembering every single second of every frame, it remembers the key moments, the new information, and the recent events, while forgetting the boring, repetitive background. It does this without needing to be retrained, meaning it can be plugged into existing AI models to make them much better at watching long, live videos.

Technical Summary: CausalMem for Efficient Streaming Video Understanding

Problem Statement

Current Multimodal Large Language Models (MLLMs) face significant challenges in streaming video understanding. The primary difficulties stem from two factors:

Unbounded Token Growth: Streaming scenarios involve potentially infinite video lengths, leading to a linear accumulation of visual tokens that exceeds the context limits and memory budgets of existing MLLMs (e.g., a two-hour video at 1 FPS can generate over 700k tokens).
Unpredictability: Unlike offline processing, streaming video lacks access to future frames or user queries in advance. This renders global redundancy-based compression methods and text-guided estimation strategies inapplicable, as they require non-causal information.

Existing solutions, such as KV cache compression or token pruning, often rely on global video access or specific query guidance, failing to provide a dynamic, fixed-budget memory mechanism that operates under strict causality.

Methodology: CausalMem

The paper proposes CausalMem, a novel, training-free approach designed to construct a dynamic but fixed-budget visual memory bank. The method operates under strict causality, meaning it processes frames sequentially without knowledge of future content. The core mechanism consists of two main components:

1. Online Semantic Basis

To estimate the redundancy of incoming visual tokens without future knowledge, CausalMem maintains an Online Semantic Basis ( $B$ ) that dynamically models the principal semantics of the observed video stream.

Initialization: The basis is initialized from the first frame's visual tokens using compact Singular Value Decomposition (SVD) to extract dominant semantic vectors.
Dynamic Update: As new frames arrive, visual tokens are projected onto the current basis. The residual (the difference between the original token and its reconstruction) represents novel information.
- Tokens with large residuals (low redundancy) are selected as candidates.
- These candidates undergo QR decomposition to extract new orthogonal basis vectors.
- The basis is updated by retaining the top- $q$ most active vectors, where activity is tracked via an exponential moving average of instantaneous activation scores. This ensures the basis adapts to evolving semantics while remaining compact.

2. Dynamic Visual Memory Bank

CausalMem maintains a fixed-size memory bank ( $M$ ) with a predefined token budget ( $b$ ).

Insertion: New visual tokens from the current frame are temporarily added to the memory bank.
Compression & Selection: If the bank exceeds the budget, tokens are scored for retention based on two factors:
1. Redundancy Score: Calculated based on the token's residual norm relative to the Online Semantic Basis. Tokens with higher residuals (more informative) are preferred.
2. Temporal Recency: A score based on the token's timestamp to ensure the memory adapts smoothly to the evolving stream.
Update: The top-scoring tokens are retained to form the updated memory bank, while others are discarded. This ensures the bank always contains the most informative and temporally relevant tokens within the fixed budget.

Key Contributions

Novel Perspective: The study approaches efficient streaming video understanding through the lens of a dynamic, fixed-budget memory bank under strict causality, addressing the limitations of existing offline or query-dependent methods.
Training-Free Mechanism: CausalMem is a plug-and-play method that requires no retraining of the underlying MLLM. It utilizes an online semantic basis to estimate redundancy and update the memory bank dynamically.
Efficiency and Preservation: The method achieves high visual token compression (e.g., >20× for hour-long videos) while preserving principal semantics, operating within a fixed storage footprint (e.g., ~82 MB for 12k tokens).

Experimental Results

The authors validated CausalMem on two representative MLLMs (LLaVA-OneVision and Qwen2.5-VL) across both streaming and offline benchmarks.

Streaming Benchmarks: On OVO-Bench and StreamingBench, CausalMem achieved average accuracy gains of +3.2% and +3.0% respectively compared to the baseline, outperforming existing streaming methods like StreamingTOM, LiveVLM, and ReKV.
Offline Benchmarks: Even when applied to offline tasks under strict causality (simulating streaming conditions), CausalMem demonstrated superior performance, outperforming specialized offline compression methods on Video-MME, MLVU, LongVideoBench, and LVBench.
Efficiency: The method significantly reduces GPU memory overhead and inference time compared to baselines that allow token counts to grow linearly. It scales better with increasing frame counts, maintaining stable inference times.
Comparison with SOTA: When applied to Qwen2.5-VL, the method achieved performance comparable to or exceeding advanced proprietary models like Gemini 1.5 Pro and GPT-4o on specific streaming benchmarks.

Significance and Claims

The paper claims that CausalMem successfully addresses the "intractable problem" of efficient streaming video understanding for existing MLLMs. Its significance lies in:

Bounded Computation: It enables the processing of arbitrarily long video streams with bounded computation and storage overhead, a critical requirement for real-world streaming applications.
Semantic Preservation: By utilizing an online semantic basis, the method maximizes information gain within a limited memory space, effectively mimicking the human brain's ability to filter redundancy while retaining salient information.
Generalizability: As a training-free approach, it can be readily integrated into various MLLMs to enhance their streaming capabilities without the need for extensive fine-tuning or architectural changes.

The authors conclude that CausalMem provides a robust solution for maintaining a compact yet informative visual memory, enabling MLLMs to handle the dynamic nature of streaming video content effectively.