CoPE-VideoLM: Leveraging Codec Primitives For Efficient Video Language Modeling

1. Problem Statement

Video Language Models (VideoLMs) aim to understand temporal dynamics in videos, but they face two critical bottlenecks:

• Context Window Constraints: Current VideoLMs rely on keyframe sampling (e.g., uniform temporal sampling) to fit videos into limited context windows. This sparse sampling often misses macro-level events and micro-level details.
• Computational Redundancy: Processing full RGB images for every sampled frame incurs massive computational overhead. Even at low frame rates (e.g., 1 FPS), consecutive frames contain high redundancy.
• Latency Issues: Encoding full images for every frame significantly increases the Time-to-First-Token (TTFT), which is detrimental to real-time applications like robotics and user experience.

Existing methods either discard temporal structure or attempt to compress dense RGB tokens after encoding, failing to leverage the inherent sparsity and structure already present in standard video codecs.

2. Methodology: CoPE-VideoLM

The authors propose CoPE-VideoLM, a framework that replaces dense RGB frame encoding with lightweight, structured representations derived directly from video codec primitives (Motion Vectors and Residuals).

Core Concepts

• Codec Primitives: Instead of decoding every frame to RGB, the model utilizes the native data from video compression (e.g., H.264, HEVC):
  • I-frames (Intra-coded): Full RGB images, processed by a standard frozen vision encoder.
  • P-frames (Predictive): Contain only the difference from the previous frame, encoded as Motion Vectors ($\tau$) and Residuals ($\delta$).
• Tokenization Strategy:
  • I-frames: Encoded into dense tokens ($M \approx 196$) via a standard Vision Encoder (e.g., SigLIP).
  • P-frames: Encoded into highly compact $\Delta$-tokens ($N \approx 8$) via a lightweight $\Delta$-Encoder.
  • Interleaving: The final input sequence to the Large Language Model (LLM) is an interleaved stream of I-frame tokens and $\Delta$-tokens, preserving the natural temporal order.

Architecture Details

1. $\Delta$-Encoder: A dual-branch lightweight transformer-based encoder:
   • Motion Branch: Processes motion vectors via an MLP and a Motion Transformer to compress spatial displacement data.
   • Residual Branch: Processes residual frames via a lightweight ResNet-18 and a Residual Transformer to capture texture/intensity changes.
   • The outputs are concatenated to form the compact $\Delta$-tokens.
2. Two-Stage Training Paradigm:
   • Stage 1 (Pre-training): The $\Delta$-Encoder is pre-trained to align codec primitives with the embedding space of the frozen vision encoder. It uses a "Reference" transformer (to warp previous frame tokens based on motion) and a "Warped" transformer (to add residuals) to reconstruct the target frame's token representation. This ensures $\Delta$-tokens are semantically compatible with RGB tokens.
   • Stage 2 (End-to-End Fine-tuning): The pre-trained $\Delta$-Encoder is integrated into the VideoLM (LLaVA-Video-7B) and fine-tuned on video instruction data. The pre-training auxiliary modules are discarded, leaving only the efficient $\Delta$-Encoder.

P-Frame Fusion

To balance temporal resolution and token count, the model can fuse $s$ consecutive P-frames. Instead of encoding every single P-frame, it encodes the cumulative change relative to a reference frame, allowing the token budget to scale naturally with video duration without hitting context limits.

3. Key Contributions

1. Codec-Aware Tokenization: A novel approach that skips redundant RGB processing for P-frames, reducing token usage by up to 93% and TTFT by up to 86%.
2. Lightweight $\Delta$-Encoder: A dual-branch architecture that effectively compresses motion and residual data into a small set of tokens ($N=8$) while maintaining alignment with image features.
3. Pre-training Strategy: A specific alignment objective that maps codec primitives to the vision encoder's space, enabling faster convergence and better performance during end-to-end fine-tuning.
4. Scalability: The method decouples frame rate from token budget, enabling open-source models to process hours of video (e.g., 8 hours at 1 FPS) within a 1M token context, a feat previously impossible for open-weight models.

4. Experimental Results

The model was evaluated on 14 diverse benchmarks covering general QA, temporal reasoning, long-form understanding, and spatial scene understanding.

• Performance vs. Efficiency:
  • Accuracy: CoPE-VideoLM matches or exceeds the performance of state-of-the-art open-source VideoLMs (like LLaVA-Video-7B) despite using a fraction of the tokens.
  • Benchmarks: It achieves SOTA among open-source models on PerceptionTest (+2.4% over baseline), NextQA, and temporal reasoning tasks like TempCompass and TOMATO.
  • Token Efficiency: In a 1 keyframe-per-GOP setting, it achieves comparable accuracy to dense sampling while using only ~6.9% of the tokens.
• Latency Gains:
  • TTFT: Reduced by 86.2% (from 2.39s to 0.33s) compared to the 64-frame baseline.
  • End-to-End Latency: Reduced by 56.1%.
• Long-Form Capability: The model can process videos up to 8 hours in length within a 1M token context, whereas standard dense sampling saturates the context window after a few minutes.
• 3D Spatial Understanding: In spatial QA tasks (ScanQA, SQA3D), the model performs competitively with specialized 3D models using only 25% of the tokens.

5. Significance and Impact

• Paradigm Shift: CoPE-VideoLM moves VideoLMs away from treating video as a sequence of independent images toward treating it as a compressed, temporally coherent stream.
• Real-World Applicability: The drastic reduction in TTFT and memory usage makes VideoLMs viable for real-time robotics and interactive applications where latency is critical.
• Resource Efficiency: By leveraging existing video compression standards, the method eliminates the need for expensive full-frame decoding and encoding, significantly lowering the carbon footprint and hardware requirements for training and inference.
• Future Scaling: As models scale to larger context windows, this approach provides a sustainable path to long-context video understanding without the quadratic compute explosion associated with dense token sequences.

In summary, CoPE-VideoLM demonstrates that codec primitives are a superior source of temporal information for VideoLMs, offering a highly efficient, scalable, and high-performance alternative to traditional dense RGB tokenization.