Technical Summary: DCVC-MB: Neural B-Frame Video Compression using State Space Models

Problem Statement

While traditional video codecs (e.g., VVC, HEVC) achieve significant compression gains through bidirectional prediction (B-frames), Neural Video Codecs (NVCs) have predominantly relied on unidirectional P-frame structures. Existing attempts to incorporate B-frames into NVCs have struggled with two primary issues:

1. Ineffective Fusion: Previous methods fail to efficiently fuse bidirectional temporal information, often underperforming compared to P-frame-only approaches.
2. Computational Complexity: Transformer-based architectures, which are capable of capturing long-range dependencies, scale quadratically ($O(N^2)$) with resolution. This leads to prohibitive memory consumption (exceeding VRAM limits) and computational costs at high resolutions (720p, 1080p), making them impractical for real-world deployment.

Methodology

The authors propose DCVC-MB (DCVC-Mamba), a neural video codec framework designed specifically for B-frame coding using State Space Models (SSMs). The architecture builds upon the DCVC-DC framework but introduces fundamental changes to enable simultaneous access to past and future temporal information.

Core Architectural Components

1. Bidirectional Feature Fusion via Mamba:

   • Unlike prior models that utilize a single Decoded Picture Buffer (DPB) from the past, DCVC-MB maintains two DPBs: one from the past frame ($X_{t-1}$) and one from the future frame ($X_{t+1}$).
   • These features are fused using a Bidirectional Mamba Block (BMB). The Mamba architecture offers linear complexity ($O(N)$) compared to the quadratic complexity of Transformers, enabling efficient processing at high resolutions.
   • The fusion process involves concatenating reference frames, embedding them via a 2D CNN, tokenizing them, and processing the sequence through the SSM. The SSM models system dynamics using a discretized Ordinary Differential Equation (ODE) with learnable parameters ($A, B, C$).
   • To capture complex bidirectional dependencies, the model employs a Spatial-First Bidirectional method, processing tokens in both forward and backward row-major scans and summing the outputs.

2. Adaptive Latent Skipping:

   • To reduce entropy coding time, the framework employs an adaptive latent skipping mechanism. Instead of a fixed threshold, it uses a relative thresholding strategy based on the standard deviation ($\sigma$) of latent elements within a frame.
   • Latents with $\sigma_i < \text{mean}(\sigma)$ are skipped (replaced by a mean value), while others are encoded. This reduces entropy coding time by approximately 9$\times$ for P-frames and 5$\times$ for B-frames without introducing asymmetric computation.

3. Canonical Position Embedding:

   • To address resolution mismatch between training (256$\times$256) and inference (up to 1080p), the authors replace standard pixel-coordinate position embeddings with Canonical Position Embeddings.
   • Spatial coordinates are normalized to a canonical image plane ($x, y \in [0, 1]$) rather than using raw pixel values. This ensures the positional harmonics remain consistent across resolutions, preventing overfitting to the training resolution.

4. Inference-Time Optimizations:

   • Open GoP: Allows the final B-frame of a Group of Pictures (GoP) to reference the I-frame of the subsequent GoP, providing temporally closer references.
   • Bidirectional Coding: Encodes odd-indexed GoPs in both forward and backward directions, selecting the result with the lower rate-distortion cost. Even-indexed GoPs use a closed configuration to maintain linear complexity.

Key Contributions

• Novel Architecture: Introduction of a bidirectional feature fusion architecture based on State Space Models (Mamba) that enables efficient B-frame coding at high resolutions where Transformer-based approaches fail due to memory constraints.
• IBP Frame Strategy: Implementation of a base neural compression model supporting Random Access (RA) video coding using IBP frame ordering.
• Efficiency Mechanisms: Development of an adaptive entropy-based latent-skipping technique to accelerate entropy coding and the integration of inference-time strategies (Open GoP and Bidirectional Coding) to enhance compression performance.
• Resolution Generalization: The use of canonical position embeddings to ensure the model generalizes effectively from low-resolution training data to high-resolution inference.

Experimental Results

The method was evaluated on standard datasets (HEVC, UVG, MCL-JCV) using RGB BT.709 conversion, comparing against VTM-19.0 (LDP and RA configurations) and state-of-the-art NVCs (DCVC-DC, DCVC-FM).

• Performance vs. Traditional Codecs: DCVC-MB achieves BD-rate reductions of up to 30.45% over VTM-19.0-LDP and 1.81% over VTM-19.0-RA (Inter-GoP=16).
• Performance vs. Neural Codecs: The model demonstrates average BD-rate reductions of 8.98% compared to prior neural video codecs (specifically DCVC-FM) and 20.19% over the stronger VTM-19.0-RA benchmark.
• Efficiency: The linear scaling of the Mamba fusion model allows the full B-frame pipeline to fit within 13.8GB VRAM at 1080p, whereas a comparable Transformer architecture exceeds 40GB.
• Ablation Studies: Experiments confirm that the Mamba fusion model accounts for the majority of BD-rate gains ($-4.55\%$), with inference-time strategies providing additive improvements. Canonical position embeddings were shown to be critical for generalization to unseen resolutions.

Significance and Claims

The paper claims that DCVC-MB represents a significant advance in neural video compression by successfully bridging the gap between the compression efficiency of bidirectional prediction and the computational feasibility of deep learning models.

The authors assert that:

1. Scalability: By leveraging Mamba, the framework achieves practical B-frame coding at high resolutions (720p, 1080p), overcoming the memory bottlenecks that have limited previous bidirectional NVCs.
2. Competitiveness: DCVC-MB is the first NVC to surpass the stronger VTM-19.0-RA configuration, demonstrating that neural approaches can now compete with or exceed the performance of leading traditional codecs in Random Access scenarios.
3. Practicality: Through adaptive latent skipping and inference-time optimizations, the method balances high compression performance with reduced encoding/decoding times, moving neural video compression closer to practical deployment.

The authors acknowledge limitations, noting that performance can degrade on specific datasets (e.g., MCL-JCV) due to domain gaps between training data (natural video) and test data (e.g., animated content with drastic motion), and that YUV420 optimization remains a challenge compared to RGB training.