💻 computer science

StreamMeCo: Long-Term Agent Memory Compression for Efficient Streaming Video Understanding

StreamMeCo is an efficient framework for streaming video understanding that reduces memory overhead and improves retrieval speed by employing edge-free minmax sampling and edge-aware weight pruning to compress memory graphs, while utilizing a time-decay retrieval mechanism to maintain or even enhance accuracy.

Original authors: Junxi Wang, Te Sun, Jiayi Zhu, Junxian Li, Haowen Xu, Zichen Wen, Xuming Hu, Zhiyu Li, Linfeng Zhang

Published 2026-04-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Junxi Wang, Te Sun, Jiayi Zhu, Junxian Li, Haowen Xu, Zichen Wen, Xuming Hu, Zhiyu Li, Linfeng Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a robot assistant living in a busy house. Every day, you watch hours of video: people cooking, kids playing, pets running around. To answer questions like "What did Mark eat for lunch?" or "Where is the red ball?", you need to remember everything you've seen.

But here's the problem: Your brain (or computer memory) is getting full.

If you try to remember every single second of every video, you eventually run out of space. Even worse, trying to search through a library of millions of memories takes so long that you can't answer the question before the user gets impatient. This is the "streaming video understanding" problem.

Enter StreamMeCo, a new system designed to be the ultimate "Memory Organizer" for these video robots. Here is how it works, using some everyday analogies:

1. The Problem: The Cluttered Attic

Imagine your memory is a giant attic filled with boxes.

The Old Way: Every time you see something, you write it down on a sticky note and throw it in the attic. After a week, you have 10,000 sticky notes. When someone asks a question, you have to dig through all 10,000 notes to find the one you need. It's slow, messy, and expensive.
The New Problem: As the video gets longer, the attic gets so full that you can't even move around in it anymore.

2. The Solution: StreamMeCo (The Smart Janitor)

StreamMeCo is like a super-smart janitor who comes in and cleans up the attic without throwing away the important stuff. It uses two main tricks to shrink the memory while keeping the answers accurate.

Trick A: The "Group Hug" (For Isolated Memories)

Some memories are just random facts with no specific person attached, like "The man ate rice" or "It was raining." These are "isolated nodes."

The Analogy: Imagine you have 100 sticky notes about "rain." They all say roughly the same thing.
What StreamMeCo does: It groups these similar notes together. Instead of keeping all 100, it picks the best representative from each group (the one closest to the center of the group) and the most unique ones that are far away from the others.
The Result: You throw away the duplicates and keep only the most diverse, important notes. It's like summarizing a 100-page report into a 1-page executive summary without losing the main points.

Trick B: The "VIP List" (For Connected Memories)

Some memories are tied to specific people or voices, like "Mark said he loves pizza." These are "connected nodes."

The Analogy: Imagine a party where some guests are just standing around chatting (low importance), while others are the hosts or VIPs (high importance).
What StreamMeCo does: It looks at who is connected to whom. If a note is about "Mark," and Mark is a main character in the video, that note gets a high "VIP score." If a note is about a background extra who never speaks, it gets a low score.
The Result: It keeps the notes about the VIPs and the most interesting conversations, while deleting the boring background chatter.

3. The Secret Sauce: The "Freshness" Filter

Even after cleaning up the attic, there's a risk: you might forget what happened just now because you focused too much on the old stuff.

The Analogy: Human memory works like a fading photograph. We remember what happened 5 minutes ago very clearly, but what happened 5 years ago is blurry.
What StreamMeCo does: It adds a "Time-Decay" filter. When you ask a question, the system prioritizes memories from the recent past. It treats recent memories like fresh fruit (keep them!) and older memories like canned goods (keep them, but maybe less of them).
The Result: The robot answers questions about what just happened instantly, while still remembering the big picture from earlier in the day.

Why is this a big deal?

The researchers tested this on three difficult "exams" (datasets) involving robots, web videos, and long movies.

Before: The robot had to carry around a massive, heavy backpack of memories. It was slow and expensive.
After (With StreamMeCo): They threw away 70% of the memory (the backpack became 30% of its original size).
The Magic: Even with less memory, the robot was 1.87 times faster at finding answers and actually got 1% more accurate at answering questions!

In a Nutshell

StreamMeCo is like hiring a professional editor for your life's video diary. Instead of keeping every single frame, it:

Summarizes boring, repetitive parts.
Highlights the important people and events.
Prioritizes what happened recently.

This allows AI agents to watch hours of video, remember the important stuff, and answer questions instantly without needing a supercomputer the size of a house to do it.

1. Problem Statement

Streaming video understanding requires models to process continuous visual information and answer user queries in real-time. Current state-of-the-art approaches, such as M3-Agent, utilize a memory graph to store long-term information (e.g., face nodes, voice nodes, and text nodes summarizing visual content). This structure maintains entity consistency and information integrity over long durations.

However, as video duration increases, the memory graph scales up, leading to two critical bottlenecks:

Storage and Computation Overhead: Storing massive amounts of memory nodes incurs high costs.
Retrieval Latency: As the number of text nodes grows (often reaching thousands), the time required to retrieve relevant memory for a query increases non-linearly, hindering real-time responsiveness.

Existing compression methods (e.g., random pruning or simple clustering) often degrade accuracy significantly because they fail to distinguish between redundant information and critical semantic details, or they ignore the structural connectivity of the memory graph.

2. Methodology: StreamMeCo

The authors propose StreamMeCo, a training-free framework designed to compress memory graphs while preserving retrieval speed and accuracy. The framework consists of two main components: Dual-Branch Memory Compression and Time-Decay Memory Retrieval (TMR).

A. Dual-Branch Memory Compression

The method categorizes text nodes in the memory graph based on their connectivity to entity nodes (faces/voices) and applies specific compression strategies to each:

Edge-free Minmax Sampling (EMsampling) for Isolated Nodes:
- Target: Text nodes not connected to specific entities (e.g., "The man's favorite food is rice").
- Mechanism:
  - Clustering: Uses Spherical KMeans to group text nodes based on embedding similarity.
  - Sampling: Within each cluster, it employs a Minmax sampling strategy. It first selects the node closest to the cluster center, then iteratively selects the node farthest from the currently selected set (maximizing the minimum distance).
- Goal: To select the most representative and diverse nodes, ensuring semantic coverage without redundancy.
Edge-aware Weight Pruning (EWpruning) for Connected Nodes:
- Target: Text nodes connected to face or voice nodes (e.g., "Mark's favorite food is rice").
- Mechanism:
  - Entity Importance: Constructs a weight edge matrix between text nodes and entity nodes. The row-wise sum determines the "entity importance" of each text node.
  - Embedding Similarity: Computes pairwise similarity between text nodes to measure redundancy.
  - Fusion Score: Combines entity importance and embedding similarity (inverted to prioritize diversity) into a fusion score: $r_j = b \cdot W'_j + (1-b) \cdot S'_j$ .
- Goal: To retain nodes that are both important to specific entities and semantically distinct from one another.

B. Time-Decay Memory Retrieval (TMR)

To mitigate performance degradation caused by compression, the authors introduce a retrieval mechanism inspired by human memory decay.

Segmentation: The memory graph is divided into time segments based on timestamps.
Relevance Scoring: For a given query, the system calculates the total similarity between the query and all text nodes within each time segment.
Decay Function: Applies an exponential decay function ( $E'_j = \bar{E}_j \cdot e^{-\lambda(t-j)}$ ) to older segments, simulating the fading of human memory.
Dynamic Allocation: The number of nodes retrieved from each segment is dynamically allocated based on the decayed relevance scores. This ensures that recent critical information is prioritized while still maintaining access to relevant long-term context.

3. Key Contributions

First Industrial-Scale Compression: StreamMeCo is the first framework specifically designed for compressing industrial-level streaming video agent memory graphs.
Structural-Aware Compression: The dual-branch approach (EMsampling for isolated nodes, EWpruning for connected nodes) leverages the specific topology of the memory graph to optimize compression efficiency.
Novel Retrieval Mechanism: The TMR mechanism effectively balances long-term memory retention with recent information retrieval, significantly reducing accuracy loss during high compression ratios.
Training-Free: The framework operates without retraining the underlying Large Multimodal Models (LMMs), making it highly practical for deployment.

4. Experimental Results

The method was evaluated on three challenging benchmarks: M3-Bench-robot (robotic first-person view), M3-Bench-web (YouTube videos), and Video-MME-Long (long-form video QA).

Performance vs. Baselines: StreamMeCo significantly outperforms existing closed-source (Gemini-1.5-Pro, GPT-4o) and open-source MLLMs, as well as other streaming video models (e.g., TimeChat-Online, Flash-VStream).
Compression Efficiency:
- Under 70% memory graph compression, StreamMeCo achieves a 1.87× speedup in memory retrieval time.
- Despite the massive reduction in memory size, it delivers an average accuracy improvement of 1.0% compared to the uncompressed M3-Agent baseline.
Ablation Studies:
- Removing the TMR mechanism causes a sharp drop in accuracy when compression exceeds 50%.
- The combination of EMsampling and EWpruning yields the best results, proving that treating isolated and connected nodes differently is crucial.
- The method shows robustness across different datasets, with the most significant gains observed on M3-Bench-robot (which contains higher redundancy).

5. Significance

StreamMeCo addresses a critical bottleneck in the deployment of long-term video agents: the trade-off between memory capacity and retrieval efficiency. By demonstrating that aggressive compression (70%) can actually improve retrieval speed without sacrificing (and potentially improving) accuracy, the paper provides a viable path toward real-time, long-horizon video understanding systems.

The framework's ability to maintain entity consistency and handle streaming inputs makes it particularly valuable for applications such as autonomous driving, real-time surveillance, and live streaming analysis, where low latency and long-term context retention are paramount. Furthermore, the training-free nature of the solution allows for immediate integration into existing agent architectures.