💻 computer science

Accelerating Streaming Video Large Language Models via Hierarchical Token Compression

The paper proposes **Streaming Token Compression (STC)**, a plug-and-play hierarchical framework that accelerates streaming VideoLLMs by using a caching mechanism to reduce Vision Transformer encoding overhead and a pruning mechanism to compress visual token sequences before LLM pre-filling.

Original authors: Yiyu Wang, Xuyang Liu, Xiyan Gui, Xinying Lin, Boxue Yang, Chenfei Liao, Tailai Chen, Linfeng Zhang

Published 2026-02-12

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Yiyu Wang, Xuyang Liu, Xiyan Gui, Xinying Lin, Boxue Yang, Chenfei Liao, Tailai Chen, Linfeng Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a live, high-speed soccer match through a tiny window, and you have a personal assistant tasked with telling you everything important that happens.

The problem is that the soccer match is moving so fast that your assistant is getting overwhelmed. They are trying to write down every single blade of grass, every movement of the crowd, and every tiny twitch of a player's muscle. Because they are writing everything down, they fall behind. By the time they finish describing a goal from two minutes ago, the game has already moved on.

This paper, "Accelerating Streaming Video Large Language Models via Hierarchical Token Compression," is like giving that assistant a superpower: the ability to ignore the boring stuff and only focus on the action.

Here is how their two-part solution (STC) works using everyday analogies:

1. STC-Cacher: The "Smart Photographer"

The Problem: In a video, most frames are almost identical. If you are filming a person talking in front of a brick wall, the wall doesn't change from one second to the next. Traditional AI "re-photographs" the entire scene (the person and the wall) every single millisecond, which is a massive waste of energy.

The Solution: Think of the STC-Cacher as a photographer who uses a "stencil" technique.

Instead of taking a brand-new photo every second, they take one perfect "Reference Photo" of the scene.
For the next few seconds, they don't re-photograph the background. They just look at the old photo, identify what moved (like the player kicking the ball), and only "paint in" those new moving parts.
Result: They save a huge amount of time because they aren't constantly re-drawing the parts of the world that stay still.

2. STC-Pruner: The "Master Editor"

The Problem: Even after the photos are taken, the assistant now has a massive pile of notes. If they try to read every single note to the boss, the boss will fall asleep before they get to the important parts. This "pile of notes" is what scientists call "token sequences," and they get too heavy for the AI's brain (the LLM) to process quickly.

The Solution: The STC-Pruner acts like a high-speed news editor. Before handing the notes to the boss, the editor looks at two things:

The "Past" Anchor: "Have I already told the boss about this background?" (If yes, throw it away).
The "Present" Anchor: "Is this specific detail actually different from the rest of this frame?" (If it's just more of the same, throw it away).

The editor only keeps the "spicy" notes—the ones that are brand new and different from both the history and the current surroundings.

Result: The AI receives a "highlight reel" instead of a "raw footage dump," allowing it to respond to your questions almost instantly.

The Bottom Line

By combining these two—the Smart Photographer (who stops re-drawing the background) and the Master Editor (who only keeps the most exciting notes)—the researchers created a way for AI to "watch" live video in real-time.

It makes the AI much faster (reducing the heavy lifting by up to 45%) without making it much dumber (it keeps 99% of its accuracy). It’s the difference between an assistant who is buried in paperwork and one who gives you a perfect, real-time play-by-play.

Technical Summary: Accelerating Streaming Video Large Language Models via Hierarchical Token Compression (STC)

1. Problem Statement

Streaming Video Large Language Models (VideoLLMs) face a critical bottleneck when deployed in real-time scenarios (e.g., live sports commentary or AR glasses). The authors identify two primary sources of inefficiency:

ViT Encoding Redundancy: In streaming settings, frames arrive continuously and are often temporally similar (e.g., static backgrounds). Existing models redundantly process every frame through the Vision Transformer (ViT), leading to high latency.
LLM Prefilling Overhead: As video progresses, the accumulation of dense visual tokens creates extremely long sequences. Due to the quadratic complexity of self-attention, the LLM prefilling stage becomes a massive computational and memory bottleneck.
Causal Constraints: Existing compression methods often rely on "global" video context (the whole video) or "instruction-aware" pruning (knowing the user's question beforehand). These are inapplicable to streaming, where frames arrive incrementally and queries are unknown until after processing.

2. Methodology: Streaming Token Compression (STC)

The authors propose STC, a plug-and-play hierarchical framework that optimizes both the vision encoding and language processing stages using two orthogonal, causal modules:

A. STC-Cacher (ViT-level Acceleration)
Instead of a full forward pass for every frame, STC-Cacher uses selective recomputation to exploit temporal redundancy:

Reference Frames: Periodically (every $N$ frames), a full forward pass is performed, and intermediate representations (Key, Value, Attention, and MLP outputs) are cached.
Non-reference Frames: For subsequent frames, the model identifies "dynamic tokens" by calculating the cosine similarity between the current Key projections and the cached reference Keys.
Selective Update: Only the dynamic tokens (those with low similarity to the cache) undergo full computation for Query, Key, Value, and MLP. The static tokens are reused from the cache. A "scatter-update" mechanism then integrates these dynamic updates back into the cached feature maps.

B. STC-Pruner (LLM-level Acceleration)
To reduce the sequence length fed into the LLM without knowing the user's query, STC-Pruner uses a dual-anchor pruning strategy:

Temporal Context Anchor (TCA): The mean of the historical token buffer, representing the "past."
Spatial Context Anchor (SCA): The mean of the current frame's tokens, representing the "present/background."
Dynamics Scoring: Each token is assigned a score based on its joint dissimilarity (cosine distance) to both anchors. Tokens that are highly dissimilar to both the past and the current frame's background are deemed "novel" and "salient."
Pruning: Only the top- $k$ tokens with the highest novelty scores are retained, effectively stripping away redundant background and repetitive temporal information.

3. Key Contributions

Empirical Discovery: The paper provides a detailed analysis of the unique characteristics of streaming video, specifically the higher temporal redundancy in "online" vs. "offline" video.
Causal-Native Design: Unlike prior works, STC is designed to be query-agnostic and future-agnostic, making it truly compatible with real-time streaming.
Hierarchical Optimization: It is one of the first frameworks to jointly optimize both the Vision Encoder (ViT) and the LLM prefilling stage.
Plug-and-Play Compatibility: The framework can be integrated into existing end-to-end online models (like Dispider or StreamForest) or offline-to-online frameworks (like ReKV) without retraining.

4. Experimental Results

The framework was evaluated across multiple benchmarks (OVO-Bench, StreamingBench, EgoSchema, etc.) with impressive results:

Efficiency: On the ReKV framework, STC reduced ViT encoding latency by 24.5% and LLM prefilling latency by 45.3%.
Accuracy: Despite the heavy compression, STC maintains high semantic fidelity, retaining up to 99% of the original accuracy on the ReKV framework.
Superiority: STC outperformed existing compression methods (like ToMe, VisionZip, and VidCom2) across various streaming and long-video understanding tasks, demonstrating a better performance-efficiency trade-off.

5. Significance

This work is significant because it addresses the "real-time gap" in VideoLLMs. By moving away from the assumption that all visual information in every frame is equally important, STC provides a scalable pathway for deploying sophisticated video intelligence in latency-sensitive, real-world applications like autonomous agents, live broadcasting, and wearable technology.