🤖 machine learning

Compressed Video Aggregator: Content-driven Module for Efficient Micro-Video Recommendation

This paper introduces the Compressed Video Aggregator (CVA), a lightweight micro-video recommendation module that decouples video information from preference learning by aggregating frozen VFM embeddings and utilizing title-guided key frame selection to achieve significant reductions in training time and memory while improving recommendation performance.

Original authors: Yang Xiao, Huiyuan Chen, Kaiyuan Deng, Chao Jiang, Zinan Ling, Ruimeng Ye, Xiaolong Ma, Bo Hui

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yang Xiao, Huiyuan Chen, Kaiyuan Deng, Chao Jiang, Zinan Ling, Ruimeng Ye, Xiaolong Ma, Bo Hui

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a massive library of short movies (micro-videos), like TikTok or YouTube Shorts. Your goal is to recommend the perfect next video to every user. The problem? There are millions of videos, and each one is a long stream of moving pictures. Trying to read every single frame of every video to understand what it's about is like trying to read every single word of a million books just to write a one-sentence summary. It takes too much time, costs too much money (in computer power), and your computer runs out of memory.

This paper introduces a new tool called CVA (Compressed Video Aggregator) to solve this problem. Think of CVA as a super-efficient "book summarizer" that doesn't need to read the whole book to know the plot.

Here is how it works, broken down into simple steps:

1. The Problem: Too Much Noise

Current systems try to understand videos in two ways, both of which have flaws:

The "ID Only" method: It just looks at the video's name or ID number. It's fast, but it's like recommending a book just because you liked the author's name, without knowing what the story is about. It misses the actual content.
The "Full Video" method: It tries to watch every frame of the video. This is accurate but incredibly slow and expensive. It's like hiring a team of 100 people to read every single page of a library before you can recommend a book.

2. The Solution: The "Smart Snippet" Strategy

The authors propose a two-step process to get the best of both worlds: Semantic Resampling and Video Compression.

Step A: Semantic Resampling (The "Highlight Reel")

Instead of watching the whole video or picking random frames (like grabbing a page at random from a book), CVA uses a smart trick.

The Analogy: Imagine you have a 5-minute video. Instead of watching it all, you ask an AI assistant (using the video's Title as a clue) to find the 5 or 8 most important moments that actually explain what the video is about.
How it works: The system looks at the video's title (e.g., "Funny Cat Fails") and scans the video to find the specific frames that match that description best. It throws away the boring, repetitive parts.
The Result: You go from processing hundreds of frames to just 5 or 8 "key frames" that tell the whole story. This is like reading a perfect 5-sentence summary instead of the whole chapter.

Step B: Video Compression (The "Distillation")

Even with just 5 or 8 frames, the computer data is still too heavy to process quickly for millions of users.

The Analogy: Imagine you have 8 high-resolution photos of a cat. They are huge files. You need to shrink them down into a single, tiny icon that still looks exactly like the cat, so your computer can carry it in its pocket.
How it works: CVA takes those 8 frames and uses a special "aggregator" (a smart math module) to blend them together into one single, tiny "video token." It keeps all the important meaning (the cat is funny) but removes all the heavy data.
The Result: The system now has a tiny, lightweight "ID" for the video that actually understands the content, not just the filename.

3. The Results: Fast, Cheap, and Smart

The authors tested this on two huge datasets of short videos. Here is what they found:

Speed: Their method was 30 times faster to train than the old, heavy methods.
Memory: It used 126 times less computer memory.
Accuracy: Surprisingly, it was actually better at recommending videos than the slow, expensive methods. By focusing only on the "key frames," it avoided getting confused by the boring parts of the video.

4. What Happens if the Clues are Wrong?

The system uses video titles to find the best frames. The authors tested what happens if the title is missing, wrong, or full of gibberish.

The Finding: Even with bad titles or no titles at all, the system still worked well. It's like a detective who can solve a crime even if the witness gives a bad description; the system is robust enough to figure out the video's content on its own.

Summary

In short, CVA is a way to teach computers to understand short videos without making them "watch" the whole thing. It picks the best few seconds, shrinks them down into a tiny, smart package, and uses that to recommend videos. It's like switching from reading a whole library to reading a perfectly written summary, allowing you to recommend books instantly without losing any of the story.

Technical Summary: Compressed Video Aggregator (CVA)

Problem Statement

Micro-video recommendation faces a critical dichotomy between the need for content-aware modeling and the computational constraints of real-world deployment. Existing methods suffer from two primary limitations:

Lack of Semantic Understanding: Many approaches rely heavily on interaction signals (clicks, watch time) or static metadata (titles, covers) without modeling the video content itself. This leads to systems that overfit training data and fail to generalize semantic or temporal structures.
Computational Prohibitions: Directly leveraging raw video content via sophisticated encoders (e.g., SlowFast, VideoMAE) or Vision-Language Models (VLMs) introduces massive overhead. These methods require extensive multi-GPU training time, consume excessive GPU memory due to simultaneous frame processing, and incur slow inference latencies, making them impractical for large-scale online recommendation systems.

Methodology

The authors propose the Compressed Video Aggregator (CVA), a lightweight, content-driven module designed to decouple visual perception from preference learning. The framework operates in two distinct phases:

1. Offline Preprocessing (One-Shot)

This phase handles data reduction and feature extraction without involving the recommendation model's training loop.

Semantic Resampling (Optional): To address temporal redundancy and the inadequacy of uniform sampling, the method partitions videos into 100 clips. It utilizes a frozen Vision-Language Model (CLIP) to compute similarity between video frames and the video title, selecting the top 5–8 most semantically relevant frames as "key frames."
Frozen Feature Extraction: A frozen Visual Foundation Model (VFM), such as DINOv3, encodes these selected key frames into dense feature vectors. These features are cached, avoiding the need for backpropagation through the heavy visual backbone during training.

2. Online Training

This phase focuses on efficient aggregation and user preference modeling.

Compressed Video Aggregator (CVA): The core innovation is a transformer-based module that condenses the variable-length sequence of frame embeddings ( $N \times D$ $N \times D$ ) into a single, compact video embedding ( $1 \times D$ $1 \times D$ ).
- Encoder: The frame embeddings are first pooled via masked mean pooling and projected through a Feedforward Network (FFN) to a latent dimension.
- Self-Attention: The latent representation is replicated and passed through a stack of Transformer blocks consisting of Self-Attention and FFN layers. The authors explicitly find that Self-Attention is crucial for capturing intra-video dependencies, while Cross-Attention (common in models like Perceiver IO) adds noise and degrades performance in this specific context.
- Decoder: The aggregated latent tokens are averaged and projected back to the final embedding dimension to serve as the input for the sequential user encoder (e.g., SASRec, GRU4Rec).

Key Contributions

Lightweight Video Embeddings: The method extracts compact, reusable frame embeddings using frozen VFMs, significantly reducing raw video redundancy.
Efficient Video Recommendation Framework: By combining optional semantic resampling with the CVA module, the framework enables efficient video modeling with minimal reliance on user/item metadata.
Performance–Efficiency Trade-off: The approach achieves a strong balance, delivering consistent gains in recommendation metrics while drastically reducing computational costs compared to heavy video backbones.

Experimental Results

Experiments were conducted on two benchmarks: MicroLens and Short-Video.

Performance Gains: On MicroLens, CVA (with 8 self-attention layers) achieved an HR@10 of 10.007, outperforming heavy baselines like SlowFast-50 (9.179) and VideoMAE (8.373).
Efficiency Improvements: Compared to SlowFast-50, CVA reduced training time by approximately 30× (46,513s $\to$ 1,427s) and peak GPU memory usage by 126× (143.85GB $\to$ 1.14GB).
Resampling Efficacy: The semantic resampling strategy (using titles to select frames) consistently improved performance across all tested visual encoders, including heavy backbones and lightweight aggregators.
Robustness: The method demonstrated robustness even under noisy conditions, such as missing titles, mismatched titles, or masked text, maintaining performance close to the clean title baseline.
Ablation Insights:
- Removing Cross-Attention from Perceiver IO improved performance, suggesting it introduces noise in this context.
- Self-Attention was identified as the critical component for effective video aggregation.
- Increasing the number of resampled frames generally improved accuracy, though with diminishing returns and increased training time, while GPU memory remained stable due to the pooling mechanism.

Significance and Claims

The paper claims that CVA offers a practical solution for deploying content-aware micro-video recommendation systems at scale. By decoupling the heavy visual perception (handled offline with frozen models) from the online learning process (handled by a lightweight aggregator), the method overcomes the computational barriers that have previously prevented the widespread adoption of video-only recommendation.

The authors emphasize that their approach does not merely compress data but effectively distills semantic information. They argue that for short-video recommendation, where only a few key frames often dominate the semantic signal, a lightweight self-attention mechanism over latent tokens is superior to complex cross-attention mechanisms or full end-to-end fine-tuning of visual backbones. The work validates that high-quality video recommendations can be achieved with orders-of-magnitude reductions in resource consumption, making content-driven modeling feasible for real-world industrial applications.