Learning Compact Video Representations for Efficient Long-form Video Understanding in Large Multimodal Models

This paper proposes an end-to-end framework for efficient long-form video understanding in large multimodal models that combines an information-density-based adaptive video sampler and an autoencoder-based spatiotemporal compressor to overcome memory constraints and extract discriminative information from redundant video sequences.

The Gist

Imagine you have a giant, 3-hour movie sitting on your desk, and you want a super-smart AI assistant (a "Large Multimodal Model" or MLLM) to watch it and answer a specific question about it.

The problem? That movie is too big.
If you try to feed the entire movie to the AI, it's like trying to drink from a firehose. The AI gets overwhelmed, runs out of memory, and starts to hallucinate (make things up) because it's drowning in too much data. Most of the movie is just people sitting still, walking slowly, or staring at a wall—lots of "boring" stuff that doesn't help answer the question.

This paper introduces a clever two-step system to solve this problem, acting like a super-efficient film editor for the AI.

The Two-Step Solution

The authors built a system with two main tools:

1. The "Smart Clipper" (Adaptive Video Sampler - AVS)

The Analogy: Imagine you are a film editor trying to find the most exciting moments in a 3-hour documentary.

• The Old Way (Uniform Sampling): You grab a pair of scissors and cut a piece of film every 10 seconds, no matter what's happening. You might cut out a boring 10 seconds of a guy sleeping, then cut out a 10-second explosion, then cut out another boring scene. You waste your "editing time" on the boring stuff.
• The New Way (AVS): This tool is like a smart editor with a sixth sense. It watches the video and only cuts the frames where something actually changes. If the camera stays on a static room for 5 minutes, it skips it. The second the door opens or a character speaks, it grabs that frame.
• The Result: Instead of showing the AI 1,000 frames (most of which are identical), it shows the AI only the 20 most important frames that tell the story.

2. The "Magic Compressor" (Spatiotemporal Video Compressor - SVC)

The Analogy: Now that you have the 20 best frames, they are still high-definition, heavy files. You need to shrink them down so the AI can carry them easily, but you can't just squish them into a tiny, unrecognizable blob.

• The Old Way (Average Pooling): Imagine taking 10 photos of a cat and 10 photos of a dog, mixing them all into a blender, and serving the AI a gray smoothie. You lose the details! The AI can't tell if it's a cat or a dog anymore.
• The New Way (SVC): This is like a high-tech compression algorithm (similar to how a ZIP file works, but smarter). It learns to "summarize" the visual information. It takes the raw video data and compresses it into a tiny, dense "latent space" (a secret code).
• The Secret Sauce: They trained this compressor using a "teacher-student" method. The compressor tries to shrink the video, and a "decoder" tries to rebuild the original video from the shrinkage. If the decoder fails to rebuild the picture, the compressor knows it threw away too much info and tries again. This ensures the AI gets a tiny file that still holds all the crucial details.

How It All Works Together

1. Input: You give the system a 2-hour video.
2. Step 1 (The Clipper): The "Smart Clipper" scans the video, ignores the boring parts, and picks out only the key moments where the action happens.
3. Step 2 (The Compressor): The "Magic Compressor" takes those key moments and shrinks them down by 64 times. It turns a massive pile of visual data into a tiny, efficient package.
4. Step 3 (The AI): The AI (the Large Language Model) receives this tiny, high-quality package. Because the data is so efficient, the AI can "read" the whole 2-hour video in its head without getting a headache or running out of memory.

Why Is This a Big Deal?

• Efficiency: The system uses 80% fewer visual tokens (data units) than previous state-of-the-art models. It's like getting the same answer from a library using only a single index card instead of reading every book.
• Accuracy: Because it doesn't get overwhelmed by boring data, it answers questions better. In tests, it beat other top models on benchmarks like EgoSchema and PerceptionTest.
• No "Hallucinations": By preserving the discriminative information (the stuff that actually matters) and throwing away the noise, the AI is less likely to make up facts.

The Bottom Line

This paper teaches us that to understand a long video, you don't need to show the AI everything. You just need to show it the right things, in the smallest possible package. It's the difference between handing someone a 500-page novel and handing them a perfectly written 5-page summary that captures the soul of the story.

Technical Summary

1. Problem Statement

Long-form video understanding (spanning tens of minutes to hours) faces significant bottlenecks when integrated with Large Multimodal Models (MLLMs). The core challenges are:

• Redundancy: Video sequences contain massive amounts of redundant visual information, leading to an overwhelming number of visual tokens.
• Computational Constraints: Transformer-based LLMs have quadratic complexity relative to input length. Processing hours of video exceeds memory and token budgets.
• Inefficiency of Current Methods:
  • Clip-level Captioning: Converting clips to text early loses low-level visual details and accumulates hallucinations.
  • Uniform Pooling: Simple average pooling treats all frames equally, causing information distortion when dissimilar frames are merged.
  • Text-Guided Compression: Methods relying on video-text pairs for training are data-hungry and difficult to scale.

2. Methodology

The authors propose an end-to-end schema comprising two novel components: an Adaptive Video Sampler (AVS) and an Autoencoder-based Spatiotemporal Video Compressor (SVC), integrated with an MLLM (QWen2).

A. Adaptive Video Sampler (AVS)

Instead of uniform sampling, AVS selects frames based on information density.

• Mechanism: It utilizes a shot boundary detection module to identify content changes (transitions between scenes/shots).
• Process:
  1. A shot detector outputs confidence scores for content changes per frame.
  2. Non-Maximum Suppression (NMS) filters redundant detections.
  3. The top-$k$ frames with the highest confidence scores (indicating dynamic moments or scene changes) are selected.
• Goal: To capture discriminative information while discarding static, redundant frames, effectively acting as a "smart" temporal subsampling strategy.

B. Autoencoder-based Spatiotemporal Video Compressor (SVC)

This module compresses the visual features of the sampled frames into a compact latent space, achieving a 64x compression ratio (e.g., 4x temporal, 4x spatial, 4x spatial).

• Architecture: A lightweight, convolution-based autoencoder (Encoder $C$ + Decoder $D$) using cascaded 3D convolutional residual blocks.
• Training Strategy (Video-Only): Unlike previous methods requiring video-text pairs, the SVC is pre-trained using video-only data via a reconstruction loss ($L_{rec}$), minimizing the difference between original and reconstructed features.
• Residual Latent Space Constraint: To prevent the compressed latent space from developing "holes" or gaps (which causes alignment failure with the LLM), the authors introduce a constraint. The latent representation $h$ is defined as the sum of the compressed features and the 3D average pooled features of the input:
  $$h = \mathcal{C}(f) + \text{avgpool}_{3D}(X)$$
  This forces the compressor to learn only the residual (lost) information relative to the average pool, ensuring the latent space aligns with the LLM's expected feature distribution without requiring stochastic constraints like VAEs.

C. Integration

The pipeline flows as: Video $\to$ AVS (Frame Selection) $\to$ Visual Encoder $\to$ SVC (Compression) $\to$ Projector $\to$ LLM. The final output is generated by the LLM based on the highly compressed, discriminative visual tokens.

3. Key Contributions

1. Novel Schema: Introduced a holistic pipeline combining information-density-based adaptive sampling and autoencoder-based compression, specifically designed for long-form video.
2. High Compression Efficiency: Achieved a 64x compression ratio (reducing token count by 80% compared to SOTA) while preserving crucial discriminative information.
3. Video-Only Pre-training: Developed a compressor that can be trained without expensive video-text pairs, relying solely on reconstruction loss and a novel residual constraint.
4. State-of-the-Art Performance: Demonstrated that the method outperforms existing MLLMs and long-form video models on multiple benchmarks while using significantly fewer computational resources.

4. Experimental Results

The method was evaluated on diverse benchmarks including EgoSchema, PerceptionTest, ActivityNet-QA, NExTQA, MVBench, and MLVU.

• Performance Gains:
  • EgoSchema: Outperformed LLaVA-OV by 2.6% (62.7% vs 60.1%) while using 80% fewer visual tokens.
  • PerceptionTest: Outperformed LLaVA-OV by 3.3%.
  • ActivityNet-QA: Surpassed LLaMA-VID and MovieChat by 4.8%.
  • General Benchmarks: Achieved SOTA or comparable results on 3 out of 6 benchmarks tested, with the remaining showing competitive performance.
• Ablation Studies:
  • AVS: Showed significant improvements (up to 1%) on long videos with shot changes (e.g., MLVU) compared to uniform sampling, particularly in tasks like anomaly detection and "needle-in-a-haystack" retrieval.
  • SVC Architecture: The AE-based compressor outperformed both average pooling and Perceiver-based downsampling at the same compression ratio.
  • Pre-training: AE pre-training provided a consistent 2-4% performance boost across all benchmarks.
  • Constraints: The proposed "Residual + Average Pool" constraint significantly outperformed unconstrained training (which failed to converge) and VAE-based constraints.

5. Significance

This work addresses a critical bottleneck in the deployment of MLLMs for long-form video analysis. By decoupling the compression and sampling process from the language model and training it efficiently on video-only data, the authors provide a scalable solution that:

• Reduces Computational Cost: Enables the processing of hours-long videos on standard hardware by drastically reducing the token budget.
• Improves Information Retention: Unlike simple pooling, the learned compression preserves essential spatiotemporal features.
• Generalizes Well: The approach is robust across various video types (short clips to 2-hour videos) and task types (QA, reasoning, description).

The paper establishes a new paradigm for efficient long-context video understanding, suggesting that adaptive selection combined with learned residual compression is superior to uniform sampling and static pooling strategies.