Uni-LVC: A Unified Method for Intra- and Inter-Mode Learned Video Compression

Uni-LVC is a unified learned video compression framework that integrates intra and inter coding into a single model by conditioning inter-coding on temporal cues via a cross-attention module and a reliability-aware classifier, thereby achieving superior rate-distortion performance across low-delay and random-access scenarios while maintaining computational efficiency.

The Gist

Imagine you are trying to send a massive library of movies to a friend over a slow internet connection. You want to send them as fast as possible without the picture looking like a blurry mess. This is the job of video compression.

For a long time, we've had two different "librarians" (algorithms) for this job:

1. The Intra Librarian: Good at compressing a single, static picture (like a snapshot). They look at the picture and say, "I can shrink this by removing redundant colors."
2. The Inter Librarian: Good at compressing moving video. They look at the previous picture and say, "Hey, that tree didn't move much, so I'll just send a note saying 'move the tree 5 pixels right' instead of redrawing the whole tree."

The Problem:
Until now, these two librarians worked in separate offices. If you wanted to send a video, you had to switch between them. Worse, the "Inter Librarian" was a bit of a worrier. If the internet glitched, or if the scene suddenly changed (like a cut from a beach to a city), the Inter Librarian would keep trying to guess based on the old picture, resulting in a terrible, glitchy mess. They couldn't easily switch back to the "Intra" mode to start fresh.

The Solution: Uni-LVC
The authors of this paper built a Super Librarian called Uni-LVC. This is a single, smart system that can do both jobs perfectly, switching between them instantly without needing two different models.

Here is how Uni-LVC works, using some everyday analogies:

1. The "Smart Assistant" Approach (Unified Model)

Instead of hiring two different people, Uni-LVC is one highly trained employee who knows how to do everything.

• The Base: They started with a very strong "Intra" expert (someone great at compressing single images).
• The Twist: They taught this expert to look at the previous frame only if it's helpful. They treat video compression as "Image compression with a hint." If the hint is good, they use it. If the hint is bad, they ignore it and just compress the image normally.

2. The "Reliability Radar" (The Classifier)

This is the paper's coolest trick. Imagine you are driving and your GPS says, "Turn left."

• Old Systems: They would blindly turn left, even if you were standing in a field or the GPS signal was broken.
• Uni-LVC: It has a Reliability Radar. Before it trusts the GPS (the previous video frame), it checks the signal.
  • Is the GPS working? Yes? Great, follow the hint!
  • Did the scene just change (like a car crash or a cut to a new scene)? The radar says, "Signal unreliable!" and immediately stops using the GPS. It switches to "Manual Mode" (Intra coding) to draw the new scene from scratch.
  • Result: No more glitchy, blurry messes when the scene changes.

3. The "Two-Pronged Search" (Cross-Attention)

When Uni-LVC looks at the previous frame to find hints, it uses a special search tool called Cross-Attention. Think of it like a detective looking for a suspect in a crowd:

• Local Search (Deformable): "Is the suspect standing right next to where they were last time?" It looks closely at the immediate neighborhood, allowing for small movements (like a person walking).
• Global Search (Linear): "Did the suspect jump to the other side of the room?" It scans the whole picture quickly to find big movements (like a camera panning).
• The Magic: It combines both searches instantly. It doesn't need to build a complex 3D map of motion; it just asks the right questions and gets the answers.

4. The "Training Camp" (Multistage Training)

You can't just throw this Super Librarian into a chaotic video game and expect them to win immediately. The authors used a clever Training Camp:

1. Phase 1: Teach them to be a master of single images (Intra).
2. Phase 2: Teach them to handle simple, slow-moving videos (Low-Delay).
3. Phase 3: Teach them to handle complex, fast-moving videos with cuts (Random Access).
4. The Secret Sauce: During the later phases, they occasionally go back and practice Phase 1 and 2. This prevents the librarian from "forgetting" how to do the basics (a problem called catastrophic forgetting).

Why Does This Matter?

• One Tool for All Jobs: You don't need different software for different types of video calls or streaming. One model handles everything.
• Robustness: If your internet connection is shaky or the video has sudden cuts, Uni-LVC doesn't crash or glitch. It adapts instantly.
• Efficiency: It compresses video better than the current state-of-the-art methods (like H.266/VVC) while running just as fast on your computer.

In a nutshell: Uni-LVC is like a Swiss Army Knife for video compression. It's a single, smart tool that knows when to use a blade (temporal hints) and when to switch to a screwdriver (intra coding) based on the situation, ensuring your video always looks crisp, no matter what happens.

Technical Summary

Here is a detailed technical summary of the paper "Uni-LVC: A Unified Method for Intra- and Inter-Mode Learned Video Compression".

1. Problem Statement

Recent advances in Learned Video Compression (LVC) have achieved significant rate-distortion (R-D) performance, often surpassing traditional codecs like H.266/VVC. However, existing LVC methods suffer from three critical limitations:

• Fragmentation: Most models are specialized for a single coding mode (Intra-only, Low-Delay Inter-only, or Random-Access Inter-only). This requires deploying separate models for different scenarios, complicating system integration and preventing seamless mode switching.
• Fragility to Unreliable References: Inter-coding models heavily rely on temporal references. When references are corrupted, mismatched, or when scene changes occur, performance degrades significantly because these models lack mechanisms to detect and suppress unreliable temporal cues.
• Lack of Unified Architecture: Unlike traditional hybrid codecs (e.g., VVC) that use a unified pipeline with hand-crafted tools to handle all modes, learned codecs typically lack a single architecture that can adaptively switch between Intra, Low-Delay (LD), and Random-Access (RA) modes.

2. Methodology

The authors propose Uni-LVC, a unified framework that treats inter-coding as intra-coding conditioned on temporal information. The core design philosophy is to build a robust intra-codec backbone and inject temporal cues adaptively rather than using separate motion estimation modules.

A. Unified Architecture

• Intra Backbone: The model uses a strong intra-codec backbone based on DCVC-RT, enhanced with:
  • Enhanced Depthwise Convolution (DC) Blocks: Incorporating spatial shifts and channel shuffling to improve spatio-channel mixing without extra parameters.
  • Hierarchical Progressive Context Model (HPCM): A simplified entropy model for accurate probability estimation.
  • Learned Lattice Vector Quantization (LVQ): To improve space-filling efficiency and reduce redundancy in the latent space.
  • Variable Rate Control: Utilizing learnable rate-control vectors and lattice density scaling to support a wide range of bitrates within a single model.
• Temporal Adaptation (Inter Mode): Inter-coding is achieved by injecting temporal features from a buffer into the intra backbone via a Cross-Attention Adaptation Module. This allows the model to reduce temporal redundancy without altering the underlying intra network structure.

B. Key Technical Components

1. Hybrid Cross-Attention Mechanism:
   • Deformable Neighborhood Cross-Attention (DN-CA): Captures local motion correspondence using deformable sampling neighborhoods.
   • Polarity-Aware Linear Cross-Attention (PAL-CA): Captures global temporal dependencies (e.g., large camera motion) with linear complexity by decomposing queries and keys into positive and negative parts to separate constructive and destructive correlations.
2. Reliability-Aware Classifier:
   • A lightweight classifier predicts a scalar gate $\alpha_t \in [0, 1]$ based on the current frame and the temporal feature.
   • If the temporal reference is unreliable (e.g., scene change, corruption), $\alpha_t$ approaches 0, suppressing the temporal feature and forcing the model to behave like an intra-coder.
   • This mechanism prevents performance collapse during scene cuts or reference mismatches.
3. Buffer Management:
   • Supports both unidirectional (LD) and bidirectional (RA) prediction.
   • Uses a recurrent update mechanism (LSTM-style) to maintain long-range temporal history in the buffer.
   • For RA, forward and backward features are fused into a unified representation.

C. Training Strategy

A multistage curriculum learning strategy is employed to prevent catastrophic forgetting and ensure balanced performance:

1. Stage 1 (Intra): Train the intra backbone to convergence.
2. Stage 2 (Low-Delay): Introduce temporal components with unidirectional references, using knowledge replay to maintain intra performance.
3. Stage 3 (Random-Access): Extend to bidirectional references, again using mode sampling and replay to preserve performance across all modes (AI, LD, RA).

3. Key Contributions

• Unified Single-Model Design: Uni-LVC is the first learned video codec to support All-Intra (AI), Low-Delay (LD), and Random-Access (RA) modes within a single model architecture.
• Robustness to Unreliable References: The introduction of the reliability-aware classifier allows the model to dynamically suppress temporal cues when references are poor, maintaining stability during scene changes where other LVCs fail.
• Efficient Hybrid Attention: The combination of DN-CA (local) and PAL-CA (global) provides a computationally efficient way to model complex temporal dependencies.
• State-of-the-Art Performance: The method achieves superior R-D performance while maintaining computational efficiency comparable to or better than specialized models.

4. Experimental Results

Experiments were conducted on standard benchmarks (HEVC Classes B-E, UVG, MCL-JCV) comparing against VTM 18.0 and various SOTA learned codecs (DCVC-RT, DCVC-FM, DCVC-B, BRHVC, etc.).

• Intra Coding (AI): Uni-LVC achieves a BD-Rate of -18.76% against VTM 18.0, outperforming DCVC-RT AI (-15.58%) and approaching much larger models like HPCM (-21.07%) with significantly fewer parameters (~50.5M vs. ~538M).
• Low-Delay (LD): Uni-LVC achieves -18.65% BD-Rate against VTM 18.0, outperforming DCVC-RT (-12.65%) and HyTIP (-14.75%). It demonstrates particularly strong performance on 1080p sequences.
• Random-Access (RA): Uni-LVC achieves 7.66% BD-Rate against VTM 18.0. While slightly behind BRHVC (4.88%) on average, it significantly outperforms DCVC-B (20.28%) and leads on high-resolution (1080p) sequences.
• Efficiency: Uni-LVC is significantly faster than competing RA/LD models. For example, it is ~14.9× faster to encode and ~12.0× faster to decode than BRHVC, with comparable or better R-D performance.
• Robustness: As shown in Figure 1 of the paper, during a scene change, Uni-LVC automatically suppresses temporal features (setting $\alpha_t \approx 0.1$), maintaining stable PSNR, whereas DCVC-RT suffers a sharp performance drop by relying on corrupted references.

5. Significance

Uni-LVC represents a major step toward the practical deployment of learned video compression. By unifying AI, LD, and RA modes into a single, robust model, it eliminates the need for multiple specialized codecs, simplifying system architecture. Its ability to adaptively handle unreliable temporal references addresses a critical weakness in previous LVC approaches, making it suitable for real-world scenarios with dynamic content and potential transmission errors. Furthermore, its high computational efficiency suggests it is viable for real-time applications, bridging the gap between theoretical performance and practical utility.