⚡ electrical engineering

A Self-Supervised Learning Framework for Video Encoding Complexity Clustering

This paper proposes Compression Echo Contrastive Learning (CECL), a novel self-supervised framework that clusters videos by encoding complexity using compression-induced signals as supervision, thereby outperforming existing visual encoders and achieving significant bitrate and quality savings in adaptive video streaming.

Original authors: Krishna Srikar Durbha, Hassene Tmar, Ping-Hao Wu, Ioannis Katsavounidis, Alan C. Bovik

Published 2026-06-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Krishna Srikar Durbha, Hassene Tmar, Ping-Hao Wu, Ioannis Katsavounidis, Alan C. Bovik

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a massive delivery service for video content. You have millions of different packages (videos) to send out. Some packages are light and easy to wrap (simple videos, like a static shot of a wall), while others are heavy, fragile, and full of complex shapes (fast-paced action movies or video games).

The big problem is that your current system treats all packages the same. It tries to wrap every single one with a generic "one-size-fits-all" strategy. This is inefficient: you waste time and fuel wrapping simple videos too heavily, and you don't wrap the complex ones tightly enough, leading to damage (poor quality) or wasted space (high file size).

The goal of this paper is to build a smarter system that can group videos together based on how hard they are to wrap (encode), so you can use the perfect wrapping strategy for each group.

The Old Way vs. The New Idea

The Old Way (Semantic Clustering):
Previously, people tried to group videos by what they look like. They would say, "All 'Vlogs' go in one box, and all 'Gaming' videos go in another."

The Flaw: The paper shows this doesn't work. Two "Vlogs" might look similar but have totally different wrapping needs. One might be a calm talking head (easy to wrap), while another might be a chaotic skateboarding video (hard to wrap). Grouping them together causes problems.

The Old Way (Quality Assessment):
Others tried to use models that judge how "pretty" or "clear" a video looks to a human eye.

The Flaw: A video can look perfect to a human but be a nightmare to compress. These models are like art critics; they care about beauty, not about how much tape it takes to ship the painting.

The New Way (CECL - Compression Echo Contrastive Learning):
The authors propose a new method called CECL. Instead of asking "What is this video?" or "Is this video pretty?", they ask: "How does this video react when we try to shrink it?"

They call this reaction the "Compression Echo."

The "Compression Echo" Analogy

Imagine you have a room full of different objects: a feather, a brick, a glass vase, and a rubber ball.

If you throw a feather against a wall, it barely makes a sound.
If you throw a brick, it makes a loud, sharp thud.
If you throw a glass vase, it shatters with a specific, chaotic noise.
If you throw a rubber ball, it bounces with a distinct boing.

In this paper, compression is the act of throwing the object against the wall. The Compression Echo is the sound it makes.

A video that compresses easily (like the feather) has a "quiet" echo.
A video that is hard to compress (like the brick or vase) has a "loud" or complex echo.

The authors built a computer system that listens to these "echoes." They don't need to know if the video is a cat or a car. They just need to know: Does this video sound like a feather or a brick when we try to shrink it?

How They Taught the Computer

Since they didn't have a teacher to label every video as "easy" or "hard," they used a Self-Supervised approach (teaching itself).

The Test: They took a batch of videos and tried to shrink them using random settings.
The Measurement: They measured the "error" or difference between the original video and the shrunk version. This difference is the Compression Echo.
The Grouping: They used a math trick to group videos that made similar "echoes" together.
- Video A and Video B both had a "quiet" echo? They are in the same group.
- Video C had a "loud" echo? It goes to a different group.
The Learning: The computer learned to recognize the visual patterns (textures, motion) that cause these specific echoes.

The Results

The researchers tested this new "Echo Listener" against the best existing systems (like those used by Meta and YouTube).

Better Grouping: When they asked the computer to sort videos into groups based on how hard they are to compress, the "Echo Listener" did a much better job than the systems that looked at "what the video is" or "how pretty it is."
Real-World Savings: When they actually used these groups to decide how to compress videos for streaming, they saved a significant amount of data (bitrate) while keeping the quality high. It's like finding a way to fit 20% more packages into the same truck without breaking anything.

Summary

The paper introduces a new way to organize videos for the internet. Instead of sorting them by their content (e.g., "sports" or "cooking"), they sort them by their physical reaction to being compressed. By listening to the "Compression Echo," their new AI can predict exactly how to handle a video to save money and bandwidth, outperforming previous methods that relied on human-like understanding of the video's content.

Technical Summary: A Self-Supervised Learning Framework for Video Encoding Complexity Clustering

Problem Statement

The paper addresses the challenge of Video Encoding Complexity Clustering, a critical task for adaptive video streaming. As video traffic grows, constructing Rate-Quality (RQ) curves for every individual video becomes computationally prohibitive. Industry solutions (e.g., Meta, YouTube) have attempted to cluster videos by their RQ characteristics to apply uniform encoding parameters, thereby reducing computational costs. However, existing approaches rely on hand-crafted features and lack comprehensive evaluation.

The authors identify three key limitations in current methods:

Semantic Similarity is Insufficient: Clustering videos by semantic categories (e.g., "Vlog" or "Animation") results in heterogeneous RQ curves. Videos with similar content often require vastly different encoding parameters, while videos from different categories may share similar encoding complexities.
Quality Assessment (IQA/VQA) Misalignment: Existing Image/Video Quality Assessment models are designed to measure perceptual quality deviations, not encoding complexity. Two videos with similar perceptual scores may have distinct rate-quality profiles.
Limitations of General SSL: Standard Self-Supervised Learning (SSL) methods often enforce invariance to spatial and temporal transformations (cropping, blurring). Since these transformations alter a video's encoding complexity, standard SSL representations are sub-optimal for this specific task.

Methodology: Compression Echo Contrastive Learning (CECL)

The authors propose Compression Echo Contrastive Learning (CECL), a novel self-supervised pretraining framework designed to learn representations specifically for encoding complexity.

Core Concept: The "Compression Echo"

The framework leverages the "Compression Echo"—the specific response of a video to compression—as a supervisory signal. The central premise is that videos exhibiting similar compression responses share underlying spatio-temporal characteristics relevant to encoding efficiency.

Pretraining Pipeline

Data Preparation: A batch of source videos is compressed using random encoding settings (resolution and CRF values). To mitigate I/O bottlenecks, videos are repeated in the batch, and random spatio-temporal masks are applied to both the original and compressed versions.
Compression Echo Calculation: The "Echo" is computed as the Mean Squared Error (MSE) between the masked original video and the masked compressed video. This metric isolates the susceptibility of visible spatio-temporal regions to the compression process.
Pseudo-Label Generation: The calculated Compression Echo values are clustered using Bisecting K-Means to generate pseudo-labels. Videos within the same cluster are treated as positive pairs, while those in different clusters are negative pairs. This process occurs dynamically during training based on the sampled batch and encoding settings.
Contrastive Objective: The model employs an asymmetric architecture (similar to BYOL) with a context branch (containing a predictor) and a target branch (updated via Exponential Moving Average). A contrastive loss function maximizes similarity between projections of videos sharing the same pseudo-label while minimizing similarity for different labels.
- Note: The authors use MSE for the echo calculation rather than VMAF or Bitrate to avoid circularity with the ground-truth evaluation metrics (BD-Rate/BD-VMAF).

Experimental Setup

Architecture: ViT-L (Vision Transformer Large) backbone.
Datasets:
- Pretraining: LAVIB (Large-scale Video Interpolation Benchmark).
- Training/Validation: LAVIB and OpenVid.
- Out-of-Domain (OOD) Evaluation: Inter4K and YouTube-UGC.
Baselines: The method is compared against state-of-the-art IQA/VQA models (CONTRIQUE, DOVER), image SSL models (CLIP, DINOv2), and video SSL models (Video-MAE, V-JEPA).
Evaluation Metrics: Clustering performance is measured using Adjusted Rand Index (ARI), Normalized Mutual Information (NMI), and Fowlkes-Mallows Index (FMI) against ground-truth labels generated by Meta's clustering algorithm. Real-world utility is assessed via BD-Rate and BD-VMAF savings against a Fixed Bitrate Ladder.

Key Results

Superior Clustering Performance: CECL outperformed all baselines on in-domain (LAVIB) and out-of-domain (Inter4K) datasets across all metrics. On 720p labels, CECL achieved an NMI of 0.442 (vs. 0.383 for V-JEPA) and an ARI of 0.226 (vs. 0.179 for V-JEPA).
Robustness to Resolution: While V-JEPA performed slightly better on the 1080p YouTube-UGC dataset, CECL demonstrated superior performance on the Inter4K dataset at 1080p, indicating strong generalization across resolutions.
Real-World Bitrate Savings: In real-world compression scenarios, CECL delivered significant bitrate savings:
- YouTube-UGC: Achieved a BD-Rate of -4.64% (compared to V-JEPA's -3.92% and an oracle of -5.03%).
- Inter4K: Achieved a BD-Rate of -4.71% (compared to V-JEPA's -4.07% and an oracle of -6.71%).
- These results were statistically significant ( $p < 0.05$ ) compared to other methods.
Ablation Studies: The method showed robustness to the number of clusters ( $K$ ) used during pretraining, with $K=10$ yielding optimal performance.

Significance and Claims

The paper claims that CECL provides a more robust and generalizable solution for video encoding complexity clustering than existing state-of-the-art visual encoders.

Novelty: It is the first work to systematically evaluate existing SoTA encoders for this specific problem and to introduce a self-supervised framework explicitly designed for encoding complexity rather than semantic understanding or perceptual quality.
Efficiency: The method achieves these gains with only approximately 1.19% additional compute over the pretraining cost of V-JEPA, demonstrating high efficiency.
Practical Impact: By learning representations that capture the intricate relationship between textures, motion, and encoding parameters, CECL enables more accurate video clustering. This allows for the application of uniform encoding parameters within clusters, significantly reducing the computational overhead of constructing RQ curves for large-scale video delivery.

The authors conclude that while the method is a significant step forward, it is primarily designed for encoding complexity clustering, and its applicability to other downstream tasks remains an open area for future exploration.