🤖 AI

Low-Bitrate Video Compression through Semantic-Conditioned Diffusion

The paper proposes DiSCo, a semantic video compression framework that transmits compact multimodal representations (text, degraded video, and optional sketches) to a conditional diffusion model for high-quality reconstruction, significantly outperforming traditional codecs in perceptual quality at ultra-low bitrates.

Original authors: Lingdong Wang, Guan-Ming Su, Divya Kothandaraman, Tsung-Wei Huang, Mohammad Hajiesmaili, Ramesh K. Sitaraman

Published 2026-04-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lingdong Wang, Guan-Ming Su, Divya Kothandaraman, Tsung-Wei Huang, Mohammad Hajiesmaili, Ramesh K. Sitaraman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to send a 4K movie to a friend, but your internet connection is as slow as a dial-up modem from the 1990s.

The Old Way (Traditional Compression):
Currently, video apps (like YouTube or Netflix) try to squeeze every single pixel of the movie into a tiny package. It's like trying to stuff a whole watermelon into a matchbox. When you force it in, the watermelon gets crushed. The result? Your video looks like a blurry, blocky mess with weird flickering colors. The computer is obsessed with keeping the exact color of every pixel, but in doing so, it loses the "soul" of the scene.

The New Way (DiSCo):
The paper you shared introduces a new method called DiSCo (Diffusion with Semantic Conditioning). Instead of trying to stuff the whole watermelon in the box, DiSCo takes a smarter approach. It realizes that your brain doesn't need to see every single pixel to understand what's happening. It just needs the story.

Here is how DiSCo works, using a simple analogy:

1. The "Recipe" Instead of the "Cake"

Imagine you want to send a friend a picture of a delicious cake.

Traditional method: You take a photo of the cake, shrink it until it's tiny, and send the photo. It looks like a blurry blob.
DiSCo method: You send your friend a recipe (the text) and a rough sketch of the cake.
- The Text: "A two-tier chocolate cake with strawberries on top." (This captures the meaning).
- The Sketch: A simple line drawing of the cake's shape. (This captures the structure).
- The Rough Video: A very low-quality, blurry video of the cake moving. (This captures the motion).

You send these three tiny files. They take up almost no space.

2. The "Magic Baker" (The AI Decoder)

Your friend receives these tiny files. Instead of just displaying the blurry blob, they use a Magic Baker (a powerful AI called a Diffusion Model).

The Magic Baker reads the recipe ("chocolate cake with strawberries"), looks at the sketch, and watches the blurry video. Using its own internal knowledge of what chocolate cakes look like, it rebuilds the cake from scratch. It paints in the rich chocolate texture, adds the shiny strawberries, and makes the movement smooth.

The result? Your friend sees a beautiful, high-quality cake, even though you only sent a few lines of text and a sketch.

3. The Secret Sauce: "Token Interleaving"

Sending a recipe, a sketch, and a video separately can still be messy. DiSCo has a clever trick called Token Interleaving.

Imagine you are telling a story to a friend, but you have three different people helping you:

Person A talks about the plot.
Person B describes the scenery.
Person C describes the action.

If they all talk at once, it's chaos. If they talk one after another, it takes too long.
DiSCo's trick: They take turns speaking word by word.

"The [Plot] [Scenery] [Action] [Plot] [Scenery] [Action]..."

This ensures the story flows perfectly without repeating information. It keeps the data stream tiny and efficient.

4. The "Time Travel" Trick

Sometimes, to save space, the video only sends every 4th frame (skipping the ones in between).

Old way: The computer just leaves a black gap or repeats the same frame, making it look jerky.
DiSCo's "Forward Filling": The AI looks at the last known frame and says, "Okay, I know where the car was. I'll just guess where it is in the next few seconds based on its speed." It fills in the missing time smoothly, so the video never stutters.

Why is this a Big Deal?

It's 2 to 10 times better: At very low internet speeds, DiSCo produces videos that look 2 to 10 times better than current technology.
It understands context: If the video is of a dog, DiSCo knows what a dog looks like. If the video is of a car, it knows what a car looks like. It doesn't just guess randomly; it uses "common sense" to fill in the missing details.
It's flexible: If the video is an animation, it sends a sketch. If it's a person dancing, it sends a skeleton map. It picks the best "clues" for the specific type of video.

The Bottom Line

Traditional compression tries to photocopy the video and shrink the paper until it tears.
DiSCo describes the video and lets a smart AI re-imagine it on the other end.

It's the difference between sending a blurry photo of a sunset versus sending a text message saying "Orange sky, purple clouds, sun setting over the ocean" and letting the receiver's imagination (powered by AI) paint the most beautiful sunset they've ever seen.

1. Problem Statement

Traditional video compression codecs (both standard like H.266 and neural like DCVC) are optimized for pixel-wise fidelity. While effective at moderate bitrates, they collapse at ultra-low bitrates, producing severe artifacts such as blocking, blurring, and temporal flickering that destroy perceptual realism.

The core issue is a misalignment between optimization objectives and human perception:

Traditional View: Minimize pixel error (PSNR/SSIM).
Human View: Interpret video through semantic understanding (objects, actions, context) rather than individual pixel accuracy.

Existing semantic compression methods often fail because they either handle spatial and temporal enhancement separately, use redundant modalities, or fail to fully exploit the compactness of semantic data.

2. Methodology: The DiSCo Framework

The authors propose DiSCo (Diffusion with Semantic Conditioning), a framework that transmits only the most meaningful information and relies on generative priors (Diffusion Models) to synthesize missing details.

A. Multimodal Encoding (Sender Side)

The source video is factorized into three compact, complementary modalities:

Textual Description (Global Semantics):
- Generated by a pretrained Vision-Language Model (LLaVA).
- Captures objects, scene context, and coarse actions.
- Compressed using a lossless codec (Zlib).
Spatiotemporally Degraded Video (Appearance & Structure):
- The source video is aggressively downsampled in both space ( $D_s$ ) and time ( $D_t$ ).
- Only coarse appearance and motion cues are retained.
- Compressed using a pretrained neural codec (DCVC-RT) with low-quality settings.
Optional Motion Cues (Sketch or Pose):
- Sketch: Extracted using InformativeDrawings for content with strong contours (e.g., animations). Compressed via a finetuned DCVC-RT.
- Pose: Extracted using DWPose for human-centric videos. Keypoints are quantized and compressed via LZMA.

B. Generative Decoding (Receiver Side)

The receiver reconstructs a high-quality, temporally coherent video from the compact bitstreams using a Conditional Video Diffusion Model.

Model Architecture: Based on LTX-Video (a 13B parameter video diffusion Transformer).
Training Strategy: Uses In-Context LoRA. Only lightweight low-rank adapters are trained on top of the frozen pretrained weights, allowing efficient adaptation to the compression task.
Key Technical Innovations:
1. Temporal Forward Filling: To handle temporal downsampling (e.g., keeping only every 2nd frame), the system duplicates the last available frame to fill dropped positions before feeding it to the model. This ensures continuity in the latent space, outperforming zero-filling or no-filling strategies.
2. Token Interleaving: Instead of concatenating tokens from different modalities (which causes redundancy), the system interleaves tokens. For example, one token from the degraded video is followed by several tokens from the auxiliary modality (sketch/pose). This ensures each frame is described exclusively by one modality, reducing information redundancy and streaming overhead.
3. Conditioning: The diffusion model is conditioned on text embeddings (T5 encoder) and the interleaved visual tokens. It is trained to reconstruct clean latent tokens from noisy/degraded inputs using Rectified Flow Matching.

3. Key Contributions

Semantic Factorization: A novel framework that decomposes video into text, degraded video, and optional motion cues (sketch/pose), maximizing compactness.
Joint Spatiotemporal Generation: A unified generation process that leverages the full generative potential of diffusion models to restore both spatial details and temporal continuity simultaneously.
Token Interleaving Mechanism: A strategy to condition diffusion Transformers efficiently, eliminating semantic redundancy between modalities and avoiding extra computational overhead.
Specialized Codecs: Custom-tuned codecs for each modality (e.g., finetuned DCVC-RT for sketches) to maximize compression efficiency.
Performance Validation: Extensive experiments demonstrating significant gains over state-of-the-art baselines.

4. Experimental Results

The method was evaluated on standard datasets (HEVC-B, MCL-JCV, UVG) and specialized datasets (Anim400K for sketches, OpenHumanVid for poses).

Perceptual Quality: DiSCo outperforms traditional codecs (H.264/265/266) and neural codecs (DCVC-RT/FM) by 5–10× in perceptual metrics (LPIPS, DISTS, FID, FVD) at low bitrates.
Semantic Compression Gain: It outperforms baseline semantic compression methods by 2–3×.
Ultra-Low Bitrate Robustness: At extremely low bitrates (e.g., 0.005 BPP or even 0.00045 BPP), traditional codecs produce unrecognizable blocks or blur. DiSCo successfully reconstructs semantically accurate scenes (e.g., recognizing a Ferris wheel or a dog) based on text and sparse cues, even if the degraded video reference is barely visible.
Ablation Studies:
- Forward Filling: Reduced BD-rate by 17.52% compared to no filling.
- Text Modality: Saved over 30% bitrate compared to no text.
- Sketch/Pose: Provided additional 3–5% bitrate reduction on specific content types.
Generalization: The model trained on 512×512, 57-frame clips generalizes to higher resolutions (1440×768) and longer sequences (233 frames) without retraining (Zero-shot).

5. Significance and Impact

Paradigm Shift: DiSCo moves video compression from a "pixel preservation" paradigm to a "semantic transmission" paradigm. It accepts that at ultra-low bitrates, perfect pixel reconstruction is impossible; instead, it transmits the intent of the video and uses AI to "hallucinate" the details in a way that aligns with human perception.
Practical Application: This approach is crucial for future applications requiring extreme bandwidth efficiency, such as satellite communications, deep-space exploration, IoT video streaming, and augmented reality in low-connectivity environments.
Efficiency: Despite using a large diffusion model, the inference latency is manageable (~396ms/frame on H100), and the training is efficient via LoRA. The framework also offers a trade-off: a 2B distilled version reduces latency by ~78% with minimal quality loss.

In conclusion, DiSCo demonstrates that by leveraging generative priors and semantic conditioning, video compression can achieve perceptual quality that traditional algorithms cannot reach at ultra-low bitrates, effectively solving the "collapse" problem of current codecs.