CurveStream: Boosting Streaming Video Understanding in MLLMs via Curvature-Aware Hierarchical Visual Memory Management

CurveStream is a training-free, curvature-aware hierarchical memory management framework that mitigates out-of-memory errors and catastrophic forgetting in streaming video understanding by adaptively routing frames based on geometric feature trajectories, achieving state-of-the-art performance with over 10% gains on key benchmarks.

The Gist

Imagine you are trying to watch a live, never-ending movie on your phone, but your phone's memory is tiny. Every time a new scene plays, your phone tries to save a picture of it. If you keep saving every single frame, your phone will crash (run out of memory) or start forgetting the beginning of the movie entirely.

This is the problem CurveStream solves for Artificial Intelligence (AI).

Here is a simple breakdown of how it works, using everyday analogies:

The Problem: The "Memory Overflow"

Current AI models are like students trying to take notes on a lecture that never ends.

• The Old Way: They try to write down everything at the same speed. Eventually, their notebook fills up, they have to tear out pages, or they start forgetting what they wrote at the beginning.
• The Result: The AI gets confused, hallucinates (makes things up), or crashes because it's trying to hold too much information at once.

The Solution: CurveStream's "Smart Highlighter"

CurveStream is a new system that acts like a super-smart highlighter for the AI. Instead of saving every frame, it decides which frames are important enough to keep and which can be forgotten or summarized.

It does this using two main tricks:

1. The "Curvature" Trick (Spotting the Drama)

Imagine you are driving a car and looking at the road ahead.

• Straight Road (Low Curvature): If the road is straight and you are driving at a constant speed, nothing exciting is happening. You don't need to pay intense attention. In video terms, this is a boring scene where the camera just pans slowly.
• Sharp Turn (High Curvature): Suddenly, the road curves sharply, or a deer jumps out. This is a "high-curvature" moment. It's a sudden change, a new event, or a critical action.

CurveStream looks at the video not as a series of pictures, but as a line drawn in the air.

• When the line is straight, the AI knows it's a boring moment and can ignore it or save a blurry, low-quality version.
• When the line makes a sharp turn (high curvature), the AI knows, "Whoa, something important just happened!" and saves a crystal-clear, high-quality picture of that exact moment.

2. The "Two-Tier" Filing System

Once the AI spots a "sharp turn" (an important moment), it doesn't just save it; it organizes it into a special filing cabinet with two drawers:

• The "Clear Memory" Drawer (High Resolution): This is for the Sharp Turns. These are the critical moments (a character speaking, an object appearing, a sudden action). These are saved in full, high-definition detail so the AI can answer questions about them later.
• The "Fuzzy Memory" Drawer (Low Resolution): This is for the Straight Roads. These are the boring, in-between moments. The AI still saves them, but it shrinks them down to a tiny, blurry thumbnail. This keeps the story going without taking up much space.

Why This is a Game-Changer

Think of it like a photographer at a wedding:

• Old AI: Tries to take a photo of every single second of the ceremony. The memory card fills up instantly, and the photographer starts deleting photos randomly, maybe deleting the ring exchange by mistake.
• CurveStream: The photographer only snaps high-quality photos when the couple kisses, the rings are swapped, or the cake is cut (the "curvature"). For the rest of the time, they just take a quick, blurry snapshot to remember where they were.

The Result:

• The AI never runs out of memory.
• It never forgets the important parts of the story.
• It can watch videos that are hours long (or even infinite streams) without getting tired or confused.

The Bottom Line

CurveStream teaches AI to pay attention to the drama and ignore the boredom. By using geometry (the shape of the video's changes) to decide what to save, it allows AI to understand live, streaming video better than ever before, beating even the most expensive commercial models in tests. It's like giving the AI a brain that knows exactly when to focus and when to relax.

Technical Summary

1. Problem Statement

Multimodal Large Language Models (MLLMs) have achieved success in offline video understanding but struggle with streaming video scenarios due to two primary bottlenecks:

• Linear Token Explosion: Streaming videos are theoretically infinite, causing the number of visual tokens to grow linearly, which leads to Out-of-Memory (OOM) errors under strict GPU constraints.
• Catastrophic Forgetting & Semantic Fragmentation: Existing solutions rely on uniform sampling, low-level physical metrics (e.g., optical flow, frame differences), or passive cache eviction (FIFO). These methods often lack intrinsic semantic awareness, leading to:
  • Disruption of contextual coherence.
  • Blurring of transient but critical semantic transitions (e.g., sudden action changes).
  • Inability to distinguish between redundant background motion (e.g., camera panning) and meaningful semantic shifts.

2. Methodology: CurveStream

CurveStream is a training-free, curvature-aware, hierarchical visual memory management framework designed to operate within a fixed token budget ($N_{max}$). It consists of two core modules:

A. Curvature-Aware Scorer (CAS)

Instead of relying on simple frame differences, CAS analyzes the evolutionary dynamics of video streams in the latent feature space.

• Geometric Insight: The authors observe that high-curvature regions in the feature trajectory correspond to critical global semantic transitions (e.g., new events, viewpoint changes, action boundaries).
• Scoring Mechanism: For each frame $t$, the system calculates a Curvature Score ($CS_t$) combining:
  1. First-order Motion Variation ($M_t$): Based on cosine similarity between consecutive frames to capture motion intensity.
  2. Second-order Geometric Curvature ($C_t$): Calculated as the angular deviation between displacement vectors of adjacent time steps ($d_1 = F_{t-1} - F_{t-2}$ and $d_2 = F_t - F_{t-1}$). This measures the change in direction of the feature evolution, effectively filtering out constant-velocity motion (like smooth camera pans) while highlighting sharp semantic shifts.
  • Formula: $CS_t = M_t + \lambda C_t$

B. Hierarchical Visual Memory Management (HVMM)

HVMM uses the Curvature Score to dynamically route frames into a fixed-capacity memory bank using an online K-Sigma dynamic threshold ($g = \mu + k\sigma$).

• Online Distribution Estimation: The system maintains a running mean ($\mu_t$) and variance ($\sigma_t$) of curvature scores using an Exponential Moving Average (EMA) to adapt to non-stationary video streams.
• Dual Thresholds: Two dynamic thresholds are generated:
  • $g_1 = \mu_t + k_1\sigma_t$
  • $g_2 = \mu_t + k_2\sigma_t$ (where $k_1 < k_2$)
• Hierarchical Routing:
  1. Clear Memory ($CS_t \ge g_2$): Frames with high semantic intensity are stored at native high resolution. These capture critical events and action boundaries.
  2. Blurred Memory ($g_1 \le CS_t < g_2$): Frames representing smooth transitions are downsampled to a low resolution (e.g., 224x224) to preserve temporal causality while saving tokens.
  3. Discard ($CS_t < g_1$): Low-information redundant frames are dropped immediately.
• Eviction Policy: When the memory queue exceeds capacity, a strict First-In-First-Out (FIFO) policy removes the oldest tokens, ensuring a constant memory footprint.

3. Key Contributions

1. Geometric Discovery: The paper reveals that high-curvature regions in the latent feature manifold align precisely with critical global semantic transitions, providing a robust metric that overcomes local noise and physical motion artifacts.
2. Training-Free Framework: CurveStream introduces a novel, plug-and-play memory management module that requires no fine-tuning of the underlying MLLM. It integrates seamlessly with various architectures (e.g., Qwen-VL, LLaVA).
3. Adaptive Hierarchical Memory: By combining real-time curvature scoring with a dynamic K-Sigma threshold, the system adaptively balances semantic integrity (High-Res) and computational efficiency (Low-Res) under strict token budgets.

4. Experimental Results

The framework was evaluated on diverse benchmarks across streaming and offline settings:

• Streaming Benchmarks (SOTA Performance):
  • StreamingBench: CurveStream achieved 84.00% accuracy with Qwen2.5-VL-7B, a 10.69% absolute improvement over the baseline.
  • OVOBench: Achieved 73.48% accuracy, a 13.58% absolute gain.
  • Outperformed specialized streaming methods (e.g., FreshMem, HERMES) and proprietary models like GPT-4o and Gemini 1.5 Pro in specific tasks.
• Offline Benchmarks:
  • Demonstrated strong generalization on MVBench (short video) and VideoMME (long video), showing consistent gains over uniform sampling baselines.
• Scalability: The method showed consistent performance gains (approx. 10-12%) across models ranging from 4B to 32B parameters (Qwen3-VL series).
• Ablation Studies:
  • Confirmed that the Curvature Metric outperforms uniform sampling, cosine similarity, and optical flow.
  • Validated that the Hierarchical Memory (mixing High/Low res) is superior to 100% High-res (causes OOM/forgetting) or 100% Low-res (loss of detail).
  • Showed robustness to hyperparameter variations ($\lambda$, $k_1$, $k_2$).

5. Significance

CurveStream addresses the fundamental limitation of MLLMs in continuous video perception: the trade-off between memory capacity and semantic fidelity.

• Theoretical Impact: It shifts the paradigm from "passive eviction" to "active semantic perception" by utilizing differential geometry (curvature) to identify information density.
• Practical Impact: It enables open-source models (e.g., 7B parameters) to surpass closed-source commercial systems in streaming tasks, making real-time, long-term video understanding feasible on consumer-grade hardware without OOM errors.
• Future Applications: The framework lays the groundwork for embodied AI applications requiring real-time adaptive reasoning, such as autonomous navigation and robotic manipulation.

In summary, CurveStream provides a lightweight, mathematically grounded solution to the "infinite video" problem, ensuring that MLLMs retain the most critical visual context while discarding redundancy, thereby achieving state-of-the-art performance in streaming video understanding.