HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding

1. Problem Statement

Multimodal Large Language Models (MLLMs) have achieved significant success in offline video understanding. However, extending these capabilities to streaming video inputs remains a critical challenge. Existing models struggle to simultaneously satisfy three conflicting requirements:

1. Stable Understanding Performance: Maintaining accuracy over long, continuous video streams.
2. Real-Time Response: Providing low-latency answers to user queries as they arrive.
3. Low Memory Overhead: Operating within limited GPU memory constraints without crashing (OOM).

Current approaches generally fall into two categories, both with significant drawbacks:

• External Memory Methods: Store video content in databases and retrieve it at query time. This introduces high latency due to retrieval and prefilling, breaking the real-time flow.
• Internal Memory (KV Cache) Methods: Existing training-free methods (e.g., ReKV, LiveVLM) often rely on external CPU/disk storage or require additional retrieval steps when a query arrives, leading to significant latency. Others use coarse-grained eviction strategies (like uniform FIFO) that ignore the nuanced attention patterns of different transformer layers.

2. Methodology: HERMES

The authors propose HERMES (KV Cache as HiERarchical Memory for Efficient Streaming Video Understanding), a training-free, plug-and-play architecture that reuses the KV cache as a hierarchical memory system.

A. Mechanistic Insight: Layer-wise Attention Analysis

The core innovation stems from a mechanistic investigation of how MLLM decoder layers attend to video tokens. The authors identified three distinct stages of attention preference:

1. Shallow Layers (Sensory Memory): Exhibit a strong recency bias, focusing intensely on the most recent frames. They function as a short-lived buffer for immediate perception.
2. Deep Layers (Long-term Memory): Show a rhythmic, sparse attention pattern where attention peaks at specific intervals corresponding to frame boundaries (e.g., every 196 tokens). These layers act as "anchor tokens," storing critical frame-level semantic summaries for long-horizon understanding.
3. Middle Layers (Working Memory): Act as a transition zone, balancing recency information with frame-level semantic representations.

B. Core Components

Based on these insights, HERMES implements three key mechanisms:

1. Hierarchical KV Cache Management:

   • Shallow Layers: Token importance is calculated using an exponential forgetting curve (inspired by Ebbinghaus' memory decay), prioritizing recent tokens.
   • Deep Layers: Token importance is derived directly from attention weights relative to a generic guidance prompt. This preserves "anchor tokens" that summarize frame semantics.
   • Middle Layers: Importance is computed by interpolating between recency scores and attention weights using a layer-dependent weight.

2. Cross-Layer Memory Smoothing:

   • To prevent misalignment where a token is kept in one layer but evicted in another, HERMES propagates importance signals from deeper to shallower layers.
   • A smoothing formula combines the raw importance score of a layer with the smoothed score of the layer below it, ensuring consistent visual memory across the network.

3. Position Re-Indexing:

   • Streaming inputs cause positional indices to grow indefinitely, degrading generation quality.
   • HERMES employs Lazy Re-Indexing for streaming scenarios: it remaps positional indices to a contiguous range $[0, |M|)$ only when necessary. This preserves the positional integrity of recent tokens while avoiding the computational overhead of "eager" re-indexing at every step.

C. Efficiency Mechanism

Unlike prior methods that offload KV states to external storage and retrieve them upon query, HERMES reuses the compact KV cache directly. When a user query arrives, no auxiliary retrieval or external computation is needed, guaranteeing real-time responses.

3. Key Contributions

1. Conceptual Framework: Pioneers the conceptualization of the KV cache as a hierarchical video memory system (Sensory, Working, Long-term) based on mechanistic attention analysis.
2. Training-Free Architecture: Proposes HERMES, a method that requires no fine-tuning and can be seamlessly integrated into existing MLLMs (e.g., LLaVA-OV, Qwen2.5-VL).
3. Performance & Efficiency: Achieves up to 68% reduction in video tokens while maintaining or improving accuracy. It delivers 10× faster Time-to-First-Token (TTFT) compared to prior State-of-the-Art (SOTA) training-free methods.

4. Experimental Results

The authors evaluated HERMES on diverse streaming and offline benchmarks using models ranging from 0.5B to 32B parameters.

• Streaming Benchmarks (StreamingBench, OVO-Bench, RVS):
  • HERMES outperformed all training-free baselines and many proprietary models.
  • On StreamingBench, HERMES (based on Qwen2.5-VL-7B) achieved 79.44% accuracy with only 4K tokens, surpassing the base model by 6.13%.
  • On RVS (Real-time Video Understanding), HERMES improved accuracy by up to 11.4% over the base model with uniform sampling.
• Offline Benchmarks (VideoMME, Egoschema, MVBench):
  • HERMES achieved competitive or superior performance compared to base models and other compression methods, demonstrating its ability to handle long-context understanding even in offline settings.
• Efficiency Metrics:
  • Latency: Achieved a 10× speedup in TTFT compared to prior SOTA (e.g., StreamingTOM).
  • Memory: Maintained a constant, compact GPU memory footprint regardless of video length, with no risk of OOM errors.
  • Real-time: Consistently maintained TTFT < 30ms even as input frames increased to 512.

5. Significance

HERMES addresses a critical bottleneck in the deployment of MLLMs for real-world streaming applications (e.g., live video analysis, surveillance, real-time assistants). By transforming the KV cache from a simple buffer into a structured, hierarchical memory system, the paper demonstrates that:

• Training-free approaches can be highly effective if grounded in mechanistic understanding of model internals.
• Real-time interaction is achievable without sacrificing accuracy or relying on expensive external retrieval systems.
• Resource efficiency can be drastically improved, enabling high-performance video understanding on single GPUs with limited memory.

This work provides a new paradigm for managing long-context multimodal inputs, shifting the focus from brute-force token retention to intelligent, layer-aware memory management.