CoStream: Codec-Guided Resource-Efficient System for Video Streaming Analytics

1. Problem Statement

Video streaming analytics using Vision-Language Models (VLMs) is becoming critical for surveillance, traffic control, and industrial automation. However, deploying VLMs for continuous video streams faces severe scalability bottlenecks due to:

• Hardware Imbalance: The number of deployed cameras vastly exceeds available GPU capacity (e.g., in London, there are ~130k cameras but only ~14k GPUs).
• Redundant Computation: VLM pipelines typically use a sliding window approach to maintain temporal context. This causes significant overlap between consecutive windows, leading the system to repeatedly process identical or highly similar visual content.
• Pipeline Inefficiencies: Existing optimization methods often target only the Vision Transformer (ViT) encoder or the Large Language Model (LLM) decoder in isolation. Furthermore, many rely on expensive offline profiling or training, making them unsuitable for dynamic, real-time streams.
• Latency Breakdown: End-to-end latency is dominated by transmission, visual processing (ViT), and LLM prefilling. Current systems fail to exploit the inherent redundancy captured by video codecs during compression.

2. Methodology: CoStream Architecture

CoStream is a holistic system that treats video codec metadata (motion vectors, residuals, frame types) not just as a byproduct of compression, but as a low-cost runtime signal to guide optimization across the entire pipeline: transmission, decoding, visual processing, and LLM prefilling.

The system consists of three core components:

A. Hardware-Accelerated Single-Pass Decoding

• Mechanism: CoStream utilizes commodity GPU video engines (e.g., NVIDIA NVDEC) to decode compressed bitstreams in a single pass.
• Optimization: Instead of decoding frames repeatedly for every overlapping sliding window, CoStream decodes the stream once and buffers the results. It extracts compressed-domain metadata (motion vectors and residuals) during this single pass.
• Benefit: Eliminates redundant decoding overhead and reduces transmission costs by operating directly on compressed bitstreams.

B. Codec-Guided Token Pruning (ViT Stage)

• Mechanism: Before the ViT encoder processes a frame, CoStream analyzes the motion vectors and residuals extracted from the codec.
• Logic:
  • Regions with near-zero motion vectors and low residuals are identified as static/predictable.
  • These regions are pruned (discarded) before entering the ViT.
  • Dynamic regions (high motion/residuals) are retained.
• Implementation: The system maps block-level codec signals to the ViT patch grid. If any patch in a spatial group is dynamic, the whole group is kept. This decision is made entirely in the compressed domain, avoiding costly online feature analysis.

C. Selective Key-Value (KV) Cache Refresh (LLM Stage)

• Mechanism: In sliding-window inference, overlapping frames share visual content but exist in different temporal contexts. Naive reuse of KV caches causes "semantic drift," while full recomputation is wasteful.
• Logic: CoStream uses codec frame-type information (I-frames vs. P-frames) to drive a selective refresh strategy:
  • Anchor Tokens: Tokens derived from I-frames (stable reference points) are recomputed to ensure semantic alignment with the new window context.
  • Reuse with Correction: Tokens from overlapping P-frames are reused but undergo position correction (using Rotary Position Embeddings, RoPE) to adjust for their new absolute positions in the sequence.
• Benefit: This balances accuracy and efficiency by refreshing only drift-sensitive states while reusing stable ones.

3. Key Contributions

1. Holistic Optimization: CoStream is the first system to unify optimization across transmission, visual processing, and LLM prefilling using a single metadata extraction pass.
2. Codec-Guided Pruning: It introduces a novel method to prune ViT tokens based on compressed-domain motion vectors, eliminating the need for offline training or expensive online similarity scoring.
3. Semantic-Aware KV Reuse: It solves the "semantic drift" problem in sliding-window LLM inference by combining selective recomputation of anchor tokens with position-corrected reuse of static tokens.
4. System Implementation: The system is implemented on top of the production vLLM framework and commodity GPU hardware, demonstrating practical deployability.

4. Experimental Results

The authors evaluated CoStream on two state-of-the-art VLMs (InternVL3-14B and Qwen3-VL-32B) using the UCF-Crime dataset with a 40-second sliding window.

• Throughput & Latency:
  • Achieved up to 3× improvement in throughput (latency reduction) compared to unoptimized baselines.
  • Specifically, 2.97× speedup for InternVL3 and 1.66× for Qwen3-VL compared to a full-compute baseline.
  • Outperformed specialized baselines like Déjà Vu (ViT-only optimization) and VLCache (LLM-only optimization).
• Resource Efficiency:
  • Reduced GPU compute (FLOPs) by up to 87%.
  • Reduced the number of processed visual tokens by 85% on average compared to full computation.
• Accuracy:
  • Maintained competitive accuracy with only a 0% to 8% drop in F1 score (e.g., InternVL3 F1 dropped from 0.89 to 0.81; Qwen3-VL showed zero degradation).
  • Performance remained stable across low, medium, and high-motion video scenarios.
• Overhead:
  • The runtime overhead for token selection and cache management was negligible (~50ms per request), accounting for less than 5% of the total optimized latency.

5. Significance

CoStream represents a paradigm shift in video analytics serving by leveraging the existing infrastructure of video compression to solve AI inference bottlenecks.

• Scalability: It addresses the critical mismatch between the explosion of camera deployments and limited GPU resources, making city-scale real-time video analytics feasible without prohibitive hardware costs.
• Generalizability: The approach is model-agnostic (works with different VLM architectures) and codec-agnostic (relies on standard primitives like motion vectors found in H.264, H.265, VP9, and AV1).
• Practicality: By avoiding offline training and utilizing standard hardware decoders, CoStream offers a deployable solution for dynamic, real-world environments where content and motion patterns change constantly.

In summary, CoStream demonstrates that treating video codecs as an active part of the inference pipeline, rather than just a preprocessing step, can yield massive efficiency gains while preserving the semantic fidelity required for complex video understanding tasks.