Technical Summary: Enhanced Neural Video Representation Compression across Extreme Complexity and Quality Scales (NVRC++)

1. Problem Statement

Implicit Neural Representations (INRs) have emerged as a promising approach for video compression, offering competitive rate-distortion performance and rapid decoding by representing videos as overfitted neural networks. However, existing INR-based codecs face a critical trade-off between complexity and scalability:

• High-performance models (e.g., NVRC, HiNeRV) typically require model complexity to increase with the target bitrate or quality. This results in significant decoding computational overhead at high bitrates and prevents the use of a single model across diverse quality levels.
• Lightweight models can achieve rate scalability with a fixed complexity but suffer from degraded compression performance due to limited representational capacity and the independent coding of video patches, which fails to exploit global spatio-temporal redundancies.
• Direct deployment of high-resolution feature grids (to improve quality without increasing network depth) often leads to suboptimal performance at low bitrates and prohibitive computational/memory overhead during the optimization (overfitting) process for long video sequences.

The core challenge is the lack of a unified architecture that maintains consistent decoding complexity across a wide range of bitrates while delivering high compression performance.

2. Methodology: The NVRC++ Framework

The authors propose NVRC++, a novel INR-based video codec designed to decouple model complexity from reconstruction quality. The framework integrates four key innovations:

A. High-Resolution Feature Grids with Fixed Complexity

NVRC++ utilizes a lightweight neural network backbone paired with multiple high-resolution feature grids. Unlike previous approaches where model size scales with quality, NVRC++ uses a single model configuration (fixed network layers) to span a wide bitrate/quality range. The high-resolution grids store dynamic spatio-temporal features directly, allowing the model to represent fine details without increasing the computational cost of the decoder network.

B. In-Parameter Coding Structure & Optimization Framework

To manage the computational and memory costs of optimizing high-dimensional grids for long sequences, NVRC++ introduces an in-parameter coding structure inspired by hierarchical coding in conventional video codecs (e.g., Hierarchical-B).

• Temporal Slicing: Feature grid parameters are partitioned into temporal slices. Dependencies are modeled hierarchically, where slices at lower levels depend on reference slices at higher levels, rather than processing the entire grid tensor simultaneously.
• Unconditional Quantization & Random Sampling: To avoid the memory explosion caused by feature propagation chains in standard overfitting, the framework employs quantization step sizes that are not conditioned on previous frames. This allows for random sampling during the optimization process, where only partial parameters are updated in each step. This significantly reduces memory footprint (enabling 600-frame 1080p sequences on 24GB GPUs) while maintaining training-testing consistency.
• Feature Grid Masking: A random masking mechanism (inspired by Dropout) is applied to high-resolution grids during early training stages. This forces the model to rely on lower-resolution grids initially, preventing overfitting to fine details at low bitrates and ensuring effective exploitation of spatio-temporal redundancy across the entire bitrate range.

C. Multi-Dimensional Grid Entropy Model

To efficiently compress the high-dimensional grid parameters, an advanced entropy model is designed that leverages three types of priors:

1. Scale Priors: Lower-resolution grids condition the coding of higher-resolution grids.
2. Temporal Priors: Coded slices from previous time steps serve as conditions for subsequent slices.
3. Spatial Priors: Spatial context within the same slice is utilized.
   This model avoids the slow, sequential nature of autoregressive models by using a multi-step coding approach with masked spatial contexts, balancing compression efficiency with high coding speed.

D. HiNeRV++ Architecture

The framework employs HiNeRV++, an enhanced version of the HiNeRV network. Key modifications include:

• Multi-Reference Inputs: Instead of relying solely on interpolated features, HiNeRV++ utilizes multiple neighboring raw feature slices as input context, providing higher-fidelity encodings.
• Simplified Output: The highest-level network blocks are replaced by a single Pixel Shuffle layer to reduce inference costs at high resolutions.
• Grid Integration: Encodings from different sets of feature grids are fed into multiple stages of the network, not just the first.

3. Key Contributions

1. Scalable INR Codec: A novel architecture supporting four distinct complexity levels (from ~7k to ~360k MACs/pixel), where each level covers a wide bitrate and quality range with a fixed decoding complexity.
2. Efficient Optimization: An optimization framework enabling the training of high-dimensional grids on long sequences under strict memory constraints via in-parameter coding and random sampling.
3. Advanced Entropy Modeling: A multi-dimensional grid entropy model that exploits scale, temporal, and spatial redundancies for efficient parameter compression.
4. Enhanced Network Design: The HiNeRV++ architecture, which achieves real-time decoding and superior performance by leveraging high-resolution grids and multi-reference inputs.

4. Experimental Results

The NVRC++ framework was evaluated on the UVG and MCL-JCV datasets (Full HD sequences) against state-of-the-art conventional codecs (x265, HM, VTM), autoencoder-based neural codecs (DCVC-FM, DCVC-RT), and existing INR-based codecs (HiNeRV, NVRC, C3).

• Rate-Distortion Performance: NVRC++ variants (S1–S4) demonstrate highly scalable performance. The S3 variant matches the performance of the SOTA INR codec NVRC while decoding 7.6 times faster. The S4 variant achieves performance comparable to VTM and DCVC-RT.
• Complexity vs. Speed: Unlike NVRC and HiNeRV, whose complexity scales with bitrate, NVRC++ maintains fixed complexity for a given scale. It achieves decoding speeds up to 73.4 FPS (S3) and 162.9 FPS (S2) on an NVIDIA RTX 4090, significantly outperforming existing INR methods.
• Ablation Studies: Removing any component (high-resolution grids, masking, entropy priors, or multi-reference inputs) resulted in significant performance degradation, validating the necessity of each innovation.
• HiNeRV++ vs. HiNeRV: The new architecture converges faster and achieves higher PSNR (e.g., +2.09 dB at 300 epochs) with lower computational cost by offloading representational burden to the grids.

5. Significance and Claims

The paper claims that NVRC++ represents a significant step toward the practical deployment of INR-based video coding. By resolving the scalability bottleneck, it enables a single model configuration to serve diverse hardware constraints and quality requirements without sacrificing compression efficiency.

The authors emphasize that NVRC++ achieves a superior complexity-performance trade-off, delivering state-of-the-art performance across an unprecedented quality range at every complexity level. Crucially, it enables real-time decoding (up to 73.4 FPS) while maintaining competitive rate-distortion performance, addressing the primary limitation of previous high-performance INR codecs that were too computationally expensive for real-world application. The work demonstrates that lightweight networks combined with high-resolution feature grids and efficient optimization strategies can outperform both traditional autoencoder-based methods and heavier INR models.