⚡ electrical engineering

GVC-RT: Towards Real-Time Generative Video Compression at Ultra-Low Bitrates

GVC-RT is a real-time generative video codec that achieves ultra-low bitrate compression with state-of-the-art fidelity by employing an asymmetric architecture and lightweight de-tokenizer to eliminate inference-time bottlenecks, enabling 1080p video processing at over 55 fps.

Original authors: Tianjian Dang, Sixian Wang, Lei Luo, Guo Lu, Jincheng Dai

Published 2026-08-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tianjian Dang, Sixian Wang, Lei Luo, Guo Lu, Jincheng Dai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to send a movie to a friend, but your internet connection is so slow that you can only send a few tiny postcards per second. In the world of video compression, this is the "ultra-low bitrate" challenge. For decades, scientists have tried to squeeze video data into smaller and smaller packages. Traditional methods work like a photocopier: they try to keep every single pixel exactly the same. But when you shrink a photo too much, it turns into a blurry, blocky mess.

Recently, a new approach called "Generative Video Compression" emerged. Instead of just copying pixels, these systems act like a skilled artist. They send a few rough sketches (the "code") and tell the receiver's computer, "Hey, you know what a ship looks like? Just draw a really good ship here." This creates stunning, realistic images even with tiny amounts of data. However, there's a catch: the "artist" on the receiving end is usually a giant, slow, super-computer brain. It takes so long to draw the picture that you can't watch the movie in real-time. It's like having a master painter who takes three days to finish a single frame. The big question for researchers has been: Can we make this artistic magic happen fast enough to watch a movie as it's being sent?

This is where a new paper introduces GVC-RT, a system designed to bring this "artist" to the speed of a regular video player. The researchers, led by Tianjian Dang and colleagues, realized that the old way of doing things was too heavy. They identified three main reasons why these generative systems were so slow: the process of turning the video into "sketches" (tokenization) was too heavy, the process of making sure the sketches matched the artist's style (alignment) was too complex, and the final step of turning the sketches back into a picture (de-tokenization) was too expensive.

To fix this, they completely redesigned the system with a clever "asymmetric" approach. Think of it like a high-stakes game of telephone where the person sending the message (the encoder) is a fast, simple runner, but the person receiving it (the decoder) is a powerful, creative painter. In the old systems, the runner had to do the painter's job too, which slowed everything down. In GVC-RT, the runner just grabs the essential information and hands it off. The heavy lifting of "generative alignment"—making sure the receiver knows exactly what style to use—is moved entirely to the training phase. It's like teaching the painter to recognize the runner's specific hand signals perfectly before the game even starts, so during the actual game, they don't need to stop and ask for clarification.

Furthermore, they replaced the giant, slow painter with a "distilled" version—a lightweight artist who is just as talented but moves much faster. They trained this lightweight artist by having it copy the work of the heavy, slow master, learning to produce the same high-quality results without the massive computational cost.

The results are impressive. The team tested their system on a standard, powerful home computer graphics card (an NVIDIA RTX 4090). They found that GVC-RT could encode video at 123.1 frames per second (fps) and decode it at 55.1 fps for high-definition 1080p video. To put that in perspective, a standard movie plays at 24 or 30 fps, so this system is fast enough to handle real-time streaming with room to spare. In terms of quality, it beat the previous best model (GLC-Video) significantly, saving about 12.4% to 48.8% in data usage while looking just as good, or even better, to the human eye.

The paper suggests that by removing the heavy "tokenization" steps and moving the complex style-matching to the training phase, they have successfully bridged the gap between high-quality, artistic video compression and real-time speed. While they acknowledge that their system still relies on powerful consumer hardware, they demonstrate that real-time generative video compression is no longer just a dream for supercomputers, but a reality that can run on devices you might find in a gaming setup today.

Technical Summary: GVC-RT

Problem Statement

Recent advances in Generative Video Compression (GVC) have demonstrated the ability to achieve high reconstruction fidelity at ultra-low bitrates (< 0.02 bits per pixel) by leveraging generative priors. However, existing GVC methods typically rely on the Generative Latent Coding (GLC) paradigm, which involves heavy generative tokenizers and complex feature alignment processes (often using diffusion models) during inference. These components introduce massive computational overhead and severe latency, preventing deployment on compute-limited devices and hindering real-time applications. While traditional neural codecs like DCVC-RT support real-time processing, they often suffer from perceptual degradations (e.g., blocking artifacts, oversmoothed textures) at ultra-low bitrates. The core challenge addressed by this paper is bridging the gap between the high perceptual quality of generative codecs and the low-latency requirements of real-time video compression.

Methodology

The authors propose GVC-RT, a novel framework that systematically redesigns the GLC pipeline to eliminate inference-time bottlenecks without sacrificing compression performance. The methodology is built on three key structural innovations:

Asymmetric Generation-Aware Encoder:
Instead of employing a heavy generative tokenizer at the encoder side to map input frames into a generative latent space, GVC-RT adopts an asymmetric architecture. It utilizes a lightweight encoder (adapted from DCVC-RT) that directly learns to extract information essential for downstream generative reconstruction. This bypasses the computationally expensive explicit tokenization step during encoding.
Soft LFQ for Latent Alignment (Training-Only):
To address the feature alignment bottleneck, the framework shifts the burden of aligning the compressed latent space with the generative manifold entirely to the training phase.
- The system employs a pre-trained Lookup-Free Quantization (LFQ) tokenizer.
- During training, the compression network is supervised to reconstruct discrete tokens by minimizing the cosine distance to the LFQ codewords.
- Crucially, a "Soft Alignment" mechanism is used where the model learns the continuous latent distribution ( $l_t$ ) rather than the hard-quantized binary tokens ( $l_{tq}$ ). This mitigates temporal flickering artifacts caused by sign flips near zero boundaries.
- At inference time, this complex alignment process is entirely removed; the network naturally projects the latent into the LFQ generative manifold without needing diffusion-based alignment modules.
Lightweight De-tokenizer with Distillation:
To resolve the final latency bottleneck during decoding, the original heavy LFQ de-tokenizer is replaced with a distilled, lightweight variant.
- The architecture shifts the computational burden to a 1/8 downsampled latent space and utilizes hardware-friendly depth-wise separable convolutions.
- A latent-to-image distillation strategy is employed: a pre-trained heavy LFQ tokenizer acts as a "teacher," providing stable latent representations. A lightweight "student" de-tokenizer is trained to reconstruct images from these teacher-provided latents, effectively transferring generative capabilities while significantly reducing parameter count and inference time.

Training Strategy:
The model undergoes a progressive three-stage training process:

Stage I: Asymmetric LFQ-AE adaptation via latent-to-image distillation to stabilize the lightweight de-tokenizer.
Stage II: Direct learning of the LFQ latent distribution using the frozen tokenizer as a supervisor, optimizing for distribution alignment and reconstruction.
Stage III: Joint end-to-end fine-tuning of the entire network, incorporating adversarial losses to enhance perceptual realism and high-frequency details.

Additionally, a perception-driven asymmetric latent supervision strategy is introduced for ultra-low bitrates. This involves directional latent alignment (using cosine similarity) and implicit dimension weighting, allowing the model to learn the varying importance of latent dimensions and attenuate errors in less critical dimensions to prevent quality collapse.

Key Contributions

GVC-RT Framework: A novel generative video compression framework designed for practical real-time ultra-low bitrate coding. It eliminates heavy inference-time tokenization and complex generative alignment modules at the encoder by directly learning generative-aware compression.
Efficient Decoding Pipeline: A redesign of the decoding process that shifts generative-space alignment entirely to training via LFQ codeword supervision and replaces the heavy de-tokenizer with a distilled lightweight variant, substantially reducing decoding latency.
Performance Balance: The framework achieves a strong balance among perceptual quality, reconstruction fidelity, and coding efficiency at ultra-low bitrates.

Experimental Results

Experiments were conducted on standard benchmarks (UVG, HEVC Class B, MCL-JCV) using an NVIDIA RTX 4090 GPU.

Compression Performance: GVC-RT outperforms the previous State-of-the-Art (SOTA) model, GLC-Video. It achieves average BD-rate savings of 12.4% (DISTS) and 48.8% (LPIPS) compared to GLC-Video. It also significantly outperforms traditional codecs (H.266/VVC) and MSE-optimized neural codecs (DCVC-RT) in perceptual metrics at ultra-low bitrates.
Real-Time Performance: GVC-RT achieves 123.1 fps for encoding and 55.1 fps for decoding on 1080p video. This represents a decoding speedup of approximately 14× compared to GLC-Video (3.9 fps) and matches the encoding speed of DCVC-RT.
Qualitative Quality: Visual comparisons show that GVC-RT preserves dynamic structures and textures (e.g., ship details, tree branches) better than MSE-optimized codecs, which tend to produce blurring artifacts at low bitrates.

Significance

The paper claims that GVC-RT represents an early step toward the practical, real-time deployment of generative codecs on consumer-grade hardware. By systematically identifying and eliminating the computational bottlenecks of the GLC paradigm (heavy tokenization, complex alignment, and costly de-tokenization), the authors demonstrate that high-realism video compression at ultra-low bitrates can be achieved without the prohibitive latency previously associated with generative models. The work suggests that real-time generative coding is feasible on single consumer GPUs (e.g., RTX 4090), paving the way for broader application in bandwidth-constrained scenarios.