💻 computer science

CoVStream: Edge-Cloud Collaboration for Understanding of Long Video Streams

CoVStream is a novel edge-cloud collaborative framework that minimizes bandwidth consumption by transmitting distilled visual features and semantic captions from resource-constrained edge devices to a cloud server for entity graph integration and on-demand heavy reasoning, achieving near-baseline accuracy with an 87.6% reduction in bandwidth usage for long video stream understanding.

Original authors: Xu Liu, Guikun Chen, Zihao Yan, Kanzhi Wu, Wenguan Wang

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xu Liu, Guikun Chen, Zihao Yan, Kanzhi Wu, Wenguan Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are wearing smart glasses that record your entire day, from the moment you wake up until you go to sleep. You want these glasses to be your personal assistant, ready to answer questions like, "Where did I leave my keys?" or "What was that funny dog I saw this morning?"

The problem is that your glasses are small and have a weak battery (the Edge Device). They can't do heavy thinking on their own. If they try to send the raw video of your whole day to a super-computer in the cloud (the Cloud Server) to get answers, it would clog up the internet connection like a traffic jam, costing a fortune in data and draining the battery instantly.

Co-VStream is a new way to solve this by creating a perfect team-up between your glasses and the cloud. Think of it as a smart secretary working for a genius detective.

The Problem with Old Methods

The "Cloud-Only" Approach: Imagine your glasses sending a live, high-definition video stream of your entire life to the cloud 24/7. It's like trying to mail a library of books every single day just to ask one question. It's too heavy, too slow, and costs too much.
The "Local-Only" Approach: Imagine your glasses trying to do all the thinking themselves. It's like asking a toddler to solve a complex math problem. They just don't have the brainpower (or memory) to remember everything you did yesterday.

The Co-VStream Solution: A Two-Step Team

Co-VStream splits the work so both sides do what they are best at, without getting in each other's way.

1. The Edge (Your Glasses): The "Smart Summarizer"

Instead of sending the raw video, your glasses act as a super-efficient editor.

The Filter: As you walk around, the glasses watch the video. If you are just walking down a street for 10 minutes without anything happening, the glasses realize, "This is boring, I don't need to send this."
The Condensation: They take the video and compress it into two tiny, lightweight things:
1. Visual Highlights: A few key "snapshots" of what the scene looks like.
2. Short Notes: Simple sentences describing what happened (e.g., "A man in a blue shirt sat on a bench").
The Result: Instead of sending a 100GB video file, they send a tiny text message and a few images. This saves 87.6% of the data traffic.

2. The Cloud (The Super-Computer): The "Genius Detective"

The cloud receives these tiny summaries and does two things:

The Memory Bank: It builds a giant, organized mind-map (called an Entity Graph). It doesn't just store pictures; it connects the dots. It knows that "The man in the blue shirt" is the same person who "sat on the bench." It keeps this map updated in real-time, but it stays asleep until you ask a question.
The Detective Work: When you ask, "Where are my keys?", the cloud wakes up instantly. It doesn't re-watch the whole day. Instead, it looks at its mind-map, finds the specific note about "keys," and gives you the answer.

Why This is a Game-Changer

It's Fast: Because the cloud is only doing the heavy thinking when you actually ask a question (and not constantly processing video), it answers in about 3 seconds. That's faster than sending the whole video to the cloud and waiting.
It's Cheap on Data: By sending summaries instead of raw video, it uses 87.6% less internet bandwidth.
It Remembers Everything: Even after 24 hours of continuous video, the system's memory stays small and stable. Other methods would run out of memory after a few hours because they try to save every single frame. Co-VStream only saves the "important stuff."
It Works on Weak Devices: Your glasses don't need a super-computer inside them. They just need to be good at summarizing, which they can do easily.

The Bottom Line

Co-VStream is like having a personal assistant who takes notes on everything you see (Edge) and a genius librarian who organizes those notes perfectly (Cloud). When you ask a question, the librarian finds the answer instantly without needing to read every book in the library. This allows your smart glasses to understand your whole day without breaking the bank or running out of battery.

Technical Summary: Co-VStream

Problem Statement

The proliferation of always-on devices (e.g., AI glasses) has established long, continuous video streams as a critical modality for personalized assistance and long-term monitoring. However, existing approaches to long video understanding rely on a "sample–encode–reason" paradigm using heavy, centralized Large Multimodal Models (LMMs). These methods face a fundamental deployment dilemma on computationally constrained edge devices:

Cloud-Only Offloading: Transmitting raw video to the cloud unlocks powerful reasoning capabilities but incurs prohibitive bandwidth costs and latency.
Local-Only Processing: Executing inference on the edge preserves bandwidth but is strictly limited by hardware capacity, preventing the deployment of the heavy models required for complex reasoning.

Current solutions fail to reconcile the high reasoning capacity of the cloud with the strict bandwidth and hardware constraints of edge deployment.

Methodology: Co-VStream

Co-VStream is the first edge-cloud collaborative framework designed specifically for understanding long video streams. It rethinks the spatiotemporal redundancy inherent in video by positing that compact semantic representations can serve as sufficient surrogates for raw pixels. The system operates via a dual-end architecture:

1. Edge Node: Dual-Stream Perception and Condensation

The edge node acts as a semantic compressor, processing the video stream in parallel to minimize latency and resource contention. It employs a Dual-Condensed Perception Pipeline consisting of four operations:

Visual Feature Extraction: A lightweight vision encoder extracts features from raw frames.
Temporal Feature Condensation: An iterative merging algorithm recursively calculates cosine similarities between adjacent features, merging the most similar pairs until a target quota is reached. This distills the raw stream into condensed visual features ( $Z_{cond}$ ).
Adaptive Keyframe Selection: A clustering algorithm selects representative keyframes ( $K$ ) from the stream to filter redundancy.
Semantic Caption Generation: A lightweight language model generates simple sentence captions ( $C_{text}$ ) for the selected keyframes.

The final payload, $P_{load} = \{Z_{cond}, C_{text}\}$ , is transmitted to the cloud. To ensure real-time execution, the pipeline utilizes Hardware-Aware Parallel Scheduling, assigning dense matrix operations (extraction, captioning) to the GPU and iterative clustering/condensation tasks to the CPU.

2. Cloud Server: Decoupled Management and Reasoning

The cloud server operates via a multi-process asynchronous architecture that decouples memory maintenance from reasoning:

Memory Management (Always Active): This process continuously integrates incoming payloads.
- Global Visual Context: Condensed visual features are merged into a global visual context ( $H$ ) using a compression bank with a fixed capacity limit to prevent memory saturation.
- Entity Graph Construction: Captions are parsed into Subject-Verb-Object (SVO) triplets. A Dynamic Entity Fusion mechanism uses a Natural Language Inference (NLI) classifier to verify bidirectional entailment between new entities and a temporal candidate pool, fusing them into a single node if they are logically equivalent. This builds a structured, dynamic Entity Graph ( $G$ ).
On-Demand Reasoning (Sleep-Wake Mechanism): The heavy reasoning model remains dormant to conserve resources. Upon receiving a user query trigger from the edge, it activates to generate a response ( $R$ $R$ ).
- Graph-Augmented Reasoning: The query is vectorized and used to retrieve the top- $k$ most relevant captions from the Entity Graph. A relevant subgraph ( $G_{sub}$ ) is extracted, serialized, and fed into the LMM alongside the global visual context and the original query to generate the final response.

Key Contributions

First Edge-Cloud Framework for Long Streams: Co-VStream is the first framework to explicitly address the trade-off between bandwidth constraints and reasoning capacity in continuous video streams.
Training-Free Design: The system operates in the original feature space of LMMs, preserving semantic distribution and allowing plug-and-play integration with various models without retraining.
Dual-Condensation Pipeline: It introduces a novel method to distill raw video into compact visual features and semantic captions, drastically reducing data transmission requirements.
Decoupled Asynchronous Architecture: By separating memory management from reasoning, the system ensures that continuous graph updates do not block instantaneous user interactions, achieving low latency.

Experimental Results

The framework was evaluated on three streaming-oriented benchmarks: VideoMME-Long, LVBench, and RTV-Bench.

Accuracy vs. Bandwidth: On LVBench, Co-VStream retains 99.2% of the Cloud-Only baseline accuracy while reducing bandwidth usage by 87.6%. On VideoMME-Long, it achieves 44.11% accuracy, retaining 94.5% of the Cloud-Only performance.
Latency: Co-VStream achieves an end-to-end latency of 2.99s, which is significantly faster than the Cloud-Only baseline (4.08s) and comparable to the Local-Only baseline (2.72s).
Ultra-Long Video Robustness: In tests exceeding 6,000 seconds, Co-VStream outperforms the Cloud-Only baseline by 7.92%, demonstrating superior handling of context windows that typically degrade in standard LMMs.
Memory Scalability: In a 24-hour (90,000s) continuous stream simulation, Co-VStream stabilized at a memory footprint of 439 MB, representing a 523× reduction compared to baselines that require storing all historical frames (accumulating ~230 GB).
Hardware Efficiency: The system maintains stable accuracy across edge devices ranging from 8GB to 48GB memory, proving viability for resource-constrained mobile agents.

Significance

The paper claims that Co-VStream bridges the disparity between limited edge hardware and the computational demands of long video understanding. By decoupling edge-side semantic condensation from cloud-side memory and reasoning, it resolves the untenable compromise between bandwidth costs and reasoning capabilities. The authors posit that this work directs community attention toward the domain of always-on deployment, extending the scope of video understanding research from offline benchmarks to real-world, continuous streaming scenarios.