Real-Time Neural Video Compression with Unified Intra and Inter Coding

This paper presents a real-time neural video compression framework that unifies intra and inter coding within a single adaptive model and employs a simultaneous two-frame compression design to effectively handle disocclusion, prevent error propagation, and achieve a 12.1% BD-rate reduction over DCVC-RT while maintaining real-time performance.

The Gist

Imagine you are trying to send a long video message to a friend over a slow, spotty internet connection. You want the video to look great, but you also need it to arrive quickly without freezing.

For decades, the standard way to do this (like H.264 or H.266) has been to send the first frame of a scene in high detail, and then for the rest of the scene, just send "instructions" on how the picture changes from the previous one. It's like sending a photo of a landscape, and then just sending a note saying, "The tree moved left, the cloud moved right." This saves a ton of data.

However, Neural Video Compression (NVC) is the new, fancy AI way of doing this. It uses smart neural networks to predict the next frame even better than the old rules. But, the current best AI methods have a few big problems, which this new paper, UI2C, solves.

Here is the breakdown of the paper using simple analogies:

1. The Problem: The "Amnesia" and the "Glitch"

Current AI video compressors are great at predicting what happens next if the scene is smooth. But they have two major flaws:

• The "Scene Change" Problem: Imagine you are watching a video of a soccer game, and suddenly it cuts to a cooking show. The AI, which is used to predicting soccer balls, panics. It tries to guess the cooking show based on the soccer game, and the result is a blurry mess. Old systems handle this by forcing a "reset" (sending a full new photo), but that costs a lot of data and creates a sudden spike in internet usage.
• The "Whispering Game" Problem: In a game of "telephone," the message gets distorted as it passes from person to person. In video compression, if the AI makes a tiny mistake predicting frame 10, that mistake gets carried over to frame 11, then 12, and so on. By frame 100, the video looks terrible. To stop this, old systems periodically force a "hard reset" (like a refresh button), which again causes those annoying data spikes.

2. The Solution: The "Swiss Army Knife" Model

The authors propose a new system called UI2C (Unified Intra and Inter Coding).

The Old Way: Imagine you have two different workers.

• Worker A (The Artist): Only works on the very first frame of a scene. They are amazing at drawing from scratch but slow.
• Worker B (The Editor): Works on all the other frames. They are fast at copying and tweaking, but if the scene changes, they are terrible at drawing from scratch.
• The Flaw: When the scene changes, you have to fire Worker B and call in Worker A. This switch is slow and expensive.

The New Way (UI2C): You hire one Super-Worker.

• This worker is trained to be both an Artist and an Editor.
• If the scene is smooth, they act like an Editor, copying the previous frame with tiny tweaks (fast and efficient).
• If the scene changes (like the soccer-to-cooking cut), they instantly switch to "Artist mode" and draw the new scene from scratch without needing a manager to tell them to switch.
• The Result: No more awkward switches, no more data spikes, and the video stays clear even after a scene change.

3. The Secret Sauce: The "Two-Step Dance"

To make this Super-Worker even faster and smarter, the authors introduced a Simultaneous Two-Frame Compression trick.

The Analogy:
Imagine you are trying to describe a dance routine to a friend over the phone.

• Old Method: You describe step 1, then step 2, then step 3. By the time you get to step 3, you might have forgotten a detail from step 1.
• UI2C Method: You look at step 1 and step 2 together before you speak. Because you can see the future (step 2), you can explain step 1 much more accurately. You know exactly how the dancer is moving into the next pose.

In technical terms, the AI looks at the current frame and the next frame at the same time. This helps it understand the motion perfectly, even if the previous frames were a bit blurry. It only adds a tiny delay (waiting for one frame), but the quality boost is huge.

4. Training the AI: The "Blindfold" Exercise

How do you teach a computer to switch between "Artist" and "Editor" modes automatically?

The authors used a clever training trick. During training, they sometimes fed the AI a blank screen (like a blank canvas) instead of the previous video frame.

• This forced the AI to learn how to draw a scene from nothing (Intra-coding).
• Sometimes they fed it a "noisy" or broken version of the previous frame.
• This taught the AI: "Hey, if the reference is bad, don't try to copy it! Just draw the new scene yourself."

This means the AI learns to fix its own mistakes on the fly, without needing a human to hit a "refresh" button.

The Bottom Line

The paper shows that this new UI2C system:

1. Saves Data: It uses about 12% less data than the current best real-time AI compressor (DCVC-RT).
2. Stays Stable: It doesn't have those annoying spikes in data usage when the scene changes.
3. Fixes Itself: It stops the "whispering game" errors from ruining the whole video.
4. Runs Fast: It's still fast enough for real-time video calls and streaming.

In short: They built a video compressor that is smart enough to know when to copy-paste and when to draw from scratch, all while looking ahead to the next frame to make sure everything looks perfect. It's like upgrading from a clumsy typist who needs a spell-checker every few words to a genius writer who knows exactly what to say before they even type it.

Technical Summary

Here is a detailed technical summary of the paper "Real-Time Neural Video Compression with Unified Intra and Inter Coding" (UI2C).

1. Problem Statement

Neural Video Compression (NVC) has achieved state-of-the-art (SOTA) efficiency, with models like DCVC-RT offering real-time performance and superior compression compared to H.266/VVC. However, existing NVC schemes suffer from three critical limitations:

• Weak Intra-Coding in P-Frames: When reference information is scarce (e.g., scene changes, disocclusion, or new content), current P-frame models lack robust intra-coding capabilities. They are forced to rely on weak inter-prediction, leading to severe quality degradation.
• Error Propagation and Accumulation: In long sequences, errors in reference features accumulate over time, degrading reconstruction quality.
• Inefficient Refresh Mechanisms: To mitigate error accumulation, current SOTA models use periodic "refresh" mechanisms (reconstructing features into pixel images and re-encoding them). This approach has two major drawbacks:
  1. It discards valuable long-term temporal cues and occluded object details along with the errors.
  2. It causes sharp bitrate spikes at refresh points, risking network congestion and hindering practical deployment.
• Model Redundancy: Traditional approaches use separate models for I-frames (intra) and P-frames (inter), increasing parameter counts and complexity.

2. Methodology: The UI2C Framework

The authors propose UI2C (Unified Intra and Inter Coding), a framework that integrates intra- and inter-frame coding into a single neural network model.

A. Unified Intra- and Inter-Frame Coding

Instead of using separate models for I-frames and P-frames, UI2C employs a single spatio-temporal network capable of adaptively switching between intra and inter modes based on reference quality.

• Mechanism: For the first frame or scene changes, the model receives a "blank" signal (all zeros) passed through an adaptor to generate reference features. This forces the model to utilize its intrinsic intra-coding capacity. For subsequent frames, it uses informative reference features to prioritize inter-coding.
• Benefit: This eliminates the need for manual refresh mechanisms. The model naturally handles scene transitions and error propagation by adaptively invoking intra-coding when references are unreliable, without discarding temporal information.

B. Simultaneous Two-Frame Compression

To further exploit temporal redundancy while maintaining real-time performance, UI2C introduces a simultaneous two-frame compression technique.

• Process: Two consecutive frames ($x_t$ and $x_{t+1}$) are concatenated along the channel dimension and processed jointly by a shared encoder-decoder.
• Latency: This introduces a 1-frame latency, which is acceptable for real-time streaming (e.g., 30fps+).
• Advantages:
  • Bidirectional Redundancy: $x_t$ can utilize $x_{t+1}$ as a backward reference, compensating for missing forward references (crucial for scene changes).
  • Error Calibration: Backward references help correct errors in occluded regions or noisy propagated features.
  • Feature Consistency: Joint downsampling suppresses high-frequency variations between consecutive frames, enhancing feature-level consistency for efficient encoding.

C. Two-Frame Quantization Strategy

To optimize Rate-Distortion (RD) performance for the jointly encoded pair:

• The system assigns distinct Quantization Parameters (QP) to $x_t$ and $x_{t+1}$ based on their frame indices.
• A higher QP (lower quality) is assigned to the latter frame ($x_{t+1}$) to ensure the subsequent frame ($x_{t+2}$) has a higher-quality reference, balancing the overall chain quality.

D. Hybrid Reference Training

To train the unified model to adaptively switch modes, the authors propose a hybrid reference strategy:

• During training, the reference for the initial frame is randomly sampled from three sources:
  1. A pure blank signal (simulating intra-dominant scenarios).
  2. Ground-truth of the previous frame.
  3. A noise-corrupted version of the previous frame (simulating error-prone inter references).
• This forces the model to learn to assess reference reliability and adaptively enhance intra-coding for error correction without manual intervention.

3. Key Contributions

1. Unified Architecture: Eliminated the need for separate I-frame and P-frame models, reducing parameter count and enabling adaptive handling of scene changes and error propagation.
2. Adaptive Error Mitigation: The model naturally intercepts inter-frame error propagation by switching to intra-coding when references are poor, removing the need for inflexible, bitrate-spiking refresh mechanisms.
3. Simultaneous Two-Frame Compression: A novel technique that leverages backward references from the next frame to boost coding efficiency and robustness while maintaining real-time inference speeds (with only 1-frame latency).
4. Hybrid Training Strategy: A training method that enables the model to dynamically balance intra/inter coding based on reference quality, ensuring robustness in long sequences.

4. Experimental Results

The model was evaluated against SOTA NVCs (DCVC-RT, DCVC-FM, DCVC-DC) and the traditional VTM (H.266) on standard datasets (HEVC Class B-E, UVG, MCL-JCV).

• Compression Efficiency: UI2C outperforms the real-time SOTA DCVC-RT by an average of 12.1% BD-rate reduction. It also outperforms the non-real-time DCVC-FM by 6.8% in rate-distortion performance.
• Real-Time Performance:
  • Encoding Speed: ~65.1 fps (1920x1080).
  • Decoding Speed: ~46.1 fps.
  • This is comparable to DCVC-RT (56.8 fps enc / 51.5 fps dec) and significantly faster than DCVC-FM (1.5 fps).
• Stability:
  • Bitrate: UI2C maintains a stable bitrate without the sharp spikes associated with refresh mechanisms.
  • Quality: In scene change scenarios (e.g., Kimono1 dataset), UI2C recovers quality much faster than DCVC-RT and avoids the quality degradation seen in non-refreshed P-frames.
• Complexity: The model has fewer parameters (46.7M) compared to DCVC-RT (66.4M) when accounting for the removal of separate I-frame models, though it has slightly higher MACs per pixel due to the joint processing. However, the effective latency per frame is reduced due to the 2-frame batch processing.

5. Significance

This work represents a significant step toward practical, real-time neural video compression. By unifying intra and inter coding, the authors solve the "scene change" and "error propagation" bottlenecks that have previously required complex, non-real-time refresh mechanisms or separate models.

• Practical Deployment: The elimination of bitrate spikes and the maintenance of real-time speeds make UI2C highly suitable for live streaming and video conferencing applications where network stability and low latency are critical.
• Paradigm Shift: It challenges the traditional separation of I-frame and P-frame processing in neural codecs, demonstrating that a single, adaptive model can outperform specialized, multi-model pipelines.
• Future Impact: The success of the unified approach suggests a path toward more robust, lightweight, and efficient video compression systems that do not rely on manual intervention for long sequences.