V-CAST: Video Curvature-Aware Spatio-Temporal Pruning for Efficient Video Large Language Models

1. Problem Statement

Video Large Language Models (VideoLLMs) struggle with long-context inference due to the massive number of visual tokens generated by consecutive video frames. This creates a bottleneck in both computation (quadratic attention complexity) and memory (KV cache size).

While token compression methods exist, the authors identify two critical failure modes in existing approaches under tight token budgets:

1. Discontinuous Spatio-Temporal Coverage:
   • Uniform Allocation: Assigns equal tokens to every frame, failing to prioritize brief but critical "decisive moments" or action peaks.
   • Segment-Based/Global Uniqueness: Methods that split videos into scenes or select globally unique frames often create "budget jumps" at boundaries or over-concentrate on a few frames, leaving transitional segments (where semantic shifts occur) under-covered.
2. Misaligned Spatio-Temporal Information:
   • Advanced VideoLLMs (e.g., Qwen3-VL) use MRoPE (Multimodal Rotary Positional Embedding), which binds tokens to discrete $(t, h, w)$ coordinates.
   • Token Merging (e.g., averaging pooling) aggregates tokens, shifting them off the original grid. This breaks the one-to-one correspondence required by MRoPE, leading to significant performance degradation (e.g., HoliTom underperforms simple pruning baselines).

Core Question: How can we select a compact set of visual tokens under a fixed budget while preserving continuous and well-aligned spatio-temporal information?

---

2. Methodology: V-CAST

The authors propose V-CAST (Video Curvature-Aware Spatio-Temporal Pruning), a training-free, plug-and-play framework. It treats token compression as a semantic trajectory approximation problem.

A. Curvature-Guided Temporal Allocation

Instead of uniform or global uniqueness allocation, V-CAST allocates the token budget based on the curvature of the semantic trajectory over time.

• Semantic Trajectory: Each frame is summarized into a mean semantic vector, normalized to a unit vector on a hypersphere.
• Curvature Calculation: The method calculates the change in direction between adjacent frames ($\kappa_t = 1 - \cos(\mathbf{v}_{in}, \mathbf{v}_{out})$). High curvature indicates a sharp semantic transition (e.g., a scene change, an action burst, or a decisive moment).
• Budget Distribution: The global token budget is distributed using an entropy-regularized optimization. Frames with higher curvature receive exponentially more tokens, ensuring that rapid semantic changes are captured without collapsing the budget entirely onto a single frame.

B. Dual-Anchor Spatial Token Selection

Once the per-frame budget is determined, V-CAST selects specific tokens within each frame using a Dual-Anchor mechanism to preserve diversity and saliency without merging:

1. Contextual Diversity Anchor: Measures how much a token deviates from the frame's mean context ($1 - \cos(\mathbf{x}_{t,i}, \bar{\mathbf{x}}_t)$). This preserves unique subjects or interactions while filtering redundant background.
2. Feature Activation Anchor: Uses the $L_2$-norm of the token features as a saliency cue, favoring strongly activated regions.

• Selection: The top-$k$ tokens with the highest combined scores are selected. Crucially, pruning is used (discarding tokens) rather than merging, ensuring all retained tokens maintain their original $(t, h, w)$ coordinates, preserving alignment with MRoPE.

---

3. Key Contributions

1. Revisiting Spatio-Temporal Coverage: The paper identifies that the core bottleneck in long-video understanding is not just token count, but the continuity and alignment of information coverage. It highlights the failure of merging strategies under MRoPE.
2. V-CAST Framework: A novel, training-free pruning policy that couples:
   • Curvature-aware temporal budgeting to route capacity to semantic turns.
   • Dual-anchor spatial selection to preserve diverse visual evidence.
   • Coordinate preservation to maintain MRoPE alignment.
3. State-of-the-Art Performance & Efficiency: V-CAST achieves superior accuracy-efficiency trade-offs across multiple VideoLLM architectures (Qwen3-VL, LLaVA-OneVision, LLaVA-Video) and scales (8B to 32B/30B MoE).

---

4. Experimental Results

The method was evaluated on benchmarks including MVBench, LongVideoBench, MLVU, VideoMME, and EgoSchema.

• Accuracy Gains:

  • On Qwen3-VL-8B-Instruct with 256 frames, V-CAST outperforms the second-best method by +1.5 points on average.
  • It preserves 98.2% of the original uncompressed performance (a +1.9% margin over the second-best).
  • On Qwen3-VL-32B, it outperforms the second-best by +1.3% on average.
  • It enables stable inference on Qwen3-VL-30B-A3B (MoE) where other methods (including the vanilla model) run out of memory (OOM) at 64 frames.

• Efficiency Improvements:

  • Memory: Reduces peak GPU memory to 86.7% of the vanilla model.
  • Latency: Reduces total inference latency to 86.4% of the vanilla model.
  • Throughput: Achieves up to 1.16× higher throughput compared to the baseline.
  • Compatibility: Unlike FlashVID (which relies on attention scores and causes OOM), V-CAST is compatible with FlashAttention and does not require extra activation overhead.

• Robustness:

  • V-CAST scales effectively from 32 to 256 input frames, whereas baselines often degrade or fail (OOM) as frame counts increase.
  • It works effectively across different positional encoding schemes (MRoPE and standard RoPE).

---

5. Significance

• Solves the MRoPE Misalignment Issue: By strictly using pruning instead of merging, V-CAST solves a critical compatibility issue with modern VideoLLMs that rely on 3D positional embeddings.
• Semantic-Aware Compression: It moves beyond static or global heuristics, dynamically allocating resources to where the video semantics actually change (curvature), mimicking human attention to "decisive moments."
• Practical Deployment: As a training-free, plug-and-play module, V-CAST offers an immediate solution for deploying long-context VideoLLMs on limited hardware, enabling efficient inference for videos up to 256 frames without sacrificing performance.
• Generalizability: The curvature-aware budgeting strategy is shown to be a generic improvement that can boost other existing compression methods (like VisionZip and VidCom2) when integrated.