Technical Summary: NU-Class Net for Video Quality Enhancement

1. Problem Statement

The proliferation of video content, now constituting over 80% of global internet traffic, has created immense pressure on storage and network bandwidth. While video compression is the standard solution, achieving high compression ratios often necessitates complex algorithms (e.g., Versatile Video Coding) that significantly increase computational demands. This complexity poses a critical barrier for resource-constrained embedded systems and IoT edge nodes (such as battery-operated drones), which often lack the processing power and energy budgets to run high-quality encoders. Consequently, these devices are forced to either operate with low efficiency or transmit low-bit-rate videos that suffer from significant perceptible artifacts and quality degradation.

2. Methodology

The paper proposes NU-Class Net, a deep learning-based approach designed to enhance video quality after decoding, rather than modifying the encoding process itself. This strategy allows edge devices to utilize simplified, low-complexity encoders to generate low-bit-rate streams, shifting the computational burden of quality restoration to the decoder side, where resources are typically more abundant.

Core Architecture

NU-Class Net is built upon the U-Net architecture, a prominent AutoEncoder model, but tailored specifically for video frame enhancement. The system operates by predicting the residual difference between a compressed, low-bit-rate frame and its original high-quality counterpart. This residual is then superimposed onto the compressed frame to reconstruct a high-quality output.

Key architectural components include:

• Encoder-Decoder Structure: The model utilizes six encoder blocks and six decoder blocks.
• NU-Blocks: Custom convolutional blocks featuring 7×7 convolution kernels (expanded from the standard 3×3) to capture a larger receptive field and precise shapes. These blocks employ Instance Normalization and activation functions like LeakyReLU and CELU.
• Skip Connections: Direct connections between encoder and decoder layers preserve low-level, high-resolution feature details (texture and pixel-level information) that are often lost during downsampling.
• Residual Blocks: Eight residual blocks are integrated into the bottleneck to facilitate the training of deeper networks and mitigate vanishing gradient issues. Five additional residual blocks are placed at the output to reduce blurriness.
• Normalization: Instance Normalization is used post-convolution to normalize across channels for individual training examples, aligning with the goal of predicting channel-specific noise or dissipated information.

Variants

The authors introduce three specific configurations to address different challenges:

1. Base NU-Class Net: The standard model predicting residuals for individual frames.
2. Sequential NU-Class Net: Incorporates temporal consistency by utilizing residual results from preceding frames to predict subsequent ones. This approach aims to reduce flickering artifacts in action scenes and accelerates training convergence by approximately 30%.
3. Diffusion NU-Class Net: Leverages diffusion model principles using a three-stage process where three NU-Class Net models work sequentially to iteratively refine the frame and remove noise.

Training Strategy

• Loss Function: The model is trained using Pixel-Distance Loss (Mean Absolute Error, MAE) rather than Mean Squared Error (MSE). MAE is preferred for its robustness to outliers and better handling of small pixel-wise differences.
• Residual Learning: The network is trained to predict the residual (the difference between raw and compressed frames) rather than the full frame, simplifying the learning task.
• Dataset: A custom dataset was created from a 2-hour video, extracting frames at 6 FPS. High-quality (CRF=13) and low-quality (CRF=40) versions were generated using FFmpeg.

3. Key Contributions

• Orthogonal Enhancement: Unlike many deep learning approaches that attempt to replace or integrate into existing codecs, NU-Class Net operates as a post-processing module. It is codec-agnostic and can be applied to any standard video stream without altering the transmission protocol or encoder hardware.
• Edge-Cloud Synergy: The method enables a shift in complexity, allowing resource-constrained edge nodes to perform low-complexity encoding while offloading the heavy lifting of quality restoration to more powerful decoder-side infrastructure.
• Residual Prediction Focus: By focusing on predicting the residual difference rather than generating entire frames, the model achieves higher efficiency and faster training compared to generative models that synthesize full images.
• Generalization: The model demonstrates the ability to generalize to unseen video content by learning fine-grained, lower-level features rather than relying solely on high-level contextual information.

4. Experimental Results

The proposed models were evaluated on a test set of 320×240 resolution frames using three metrics: Pixel-Distance Loss, Peak Signal-to-Noise Ratio (PSNR), and Structural Similarity Index (SSIM).

• Compression Efficiency: The system achieved a video file size reduction of up to 17x (from ~7910 MB to ~445 MB) while maintaining the original resolution, compared to a "typical" video which offered only a ~1.7x reduction.
• Quality Improvement:
  • PSNR: All proposed models (Base, Sequential, Diffusion) successfully elevated PSNR values above the acceptable threshold of 30 dB (ranging from ~31.0 to 33.7 dB), whereas the compressed input was below 28 dB.
  • SSIM: The models improved SSIM scores from below the 0.9 threshold (0.855) to above it (ranging from 0.906 to 0.935), indicating restored structural similarity.
  • Loss Reduction: The training process resulted in a ~30% reduction in test set loss.
• Execution Time: On a V100 GPU, the Base NU-Class Net processes a single frame in 83.33 ms (24 fps). Through 16-bit fixed-point quantization, the execution time was reduced to 19.84 ms, and the model size was halved, with accuracy loss remaining within 1% of the original 32-bit model.

5. Significance and Claims

The paper positions NU-Class Net as a practical solution for the "Internet of Video Things" (IoVT). Its primary significance lies in enabling high-quality video experiences in environments with strict energy and bandwidth constraints. By allowing edge devices to transmit highly compressed streams without the computational penalty of high-fidelity encoding, the approach reduces bandwidth usage and energy consumption at the source.

The authors claim that this method provides a "proof of concept" for utilizing deep learning to decouple video quality from encoding complexity. They emphasize that the model's ability to operate post-decoding makes it a versatile tool that can be synergistically used with any modern video codec to further mitigate bit-rate requirements without requiring hardware changes at the capture node. The work suggests that future advancements in hardware acceleration could further reduce the decoder-side complexity, making this approach even more viable for widespread IoT deployment.