Exploring Real-Time Super-Resolution: Benchmarking and Fine-Tuning for Streaming Content

1. Problem Statement

Real-time super-resolution (SR) is critical for enhancing video streaming quality on bandwidth-constrained platforms (e.g., YouTube, Twitch). However, existing SR methods face two primary limitations:

• Dataset Mismatch: Most state-of-the-art (SoTA) models are trained on clean, synthetic datasets (e.g., DIV2K, Vimeo90K) where Low-Resolution (LR) images are generated via bicubic downsampling. These datasets fail to capture the specific artifacts (blockiness, blurring, ringing) introduced by real-world video compression codecs (H.264, VP9, AV1).
• Performance Gap: While models like NVIDIA VSR offer real-time speeds, they often suffer from over-smoothing and fail to restore fine textures. Conversely, high-quality models designed for compressed video are often too computationally heavy for real-time deployment (sub-30 FPS).

2. Methodology

The authors propose a three-pronged approach: a new dataset, a novel efficient model, and a systematic benchmarking framework.

A. The StreamSR Dataset

To bridge the gap between synthetic training and real-world streaming, the authors introduced StreamSR:

• Source: 5,200 diverse YouTube videos (over 10 million frames) covering 20 categories (Nature, Sports, Gaming, etc.).
• Selection Criteria: Videos were filtered for Creative Commons (CC BY 4.0) licensing and specific resolutions (360p, 720p, 1440p) to create 2× and 4× upscaling tracks.
• Quality Control: Videos were clustered using spatial/temporal information (SI/TI), bitrate, and semantic embeddings (SigLIP) to ensure diversity.
• Unique Feature: Unlike previous datasets, StreamSR provides aligned LR-HR pairs derived directly from compressed streams, preserving real compression artifacts.

B. The EfRLFN Model

The authors propose EfRLFN (Efficient Residual Lightweight Feature Network), an optimized image-based SR model designed for real-time inference. It builds upon the RLFN architecture with specific modifications:

• Activation Function: Replaces the standard ReLU with Hyperbolic Tangent (tanh) in refinement modules. Unlike ReLU, which discards negative activations, tanh preserves the sign and magnitude of features, improving gradient flow and attention map accuracy.
• Attention Mechanism: Replaces the heavy Enhanced Spatial Attention (ESA) with Efficient Channel Attention (ECA). ECA uses global average pooling and a 1×1 convolution, significantly reducing computational overhead while maintaining performance.
• Architecture: Retains the lightweight feature extractor and reconstruction blocks of RLFN but streamlines the ERLFB (Efficient Residual Local Feature Block) by removing redundant skip connections.
• Training Strategy: Introduces a Composite Loss Function to replace RLFN's complex two-stage contrastive learning:
  1. Charbonnier Loss: For robust pixel-level reconstruction.
  2. VGG Perceptual Loss: For perceptual realism using pre-trained VGG-19 features.
  3. Sobel Edge Loss: Explicitly optimizes edge sharpness to prevent blurring.
  • Result: Single-stage end-to-end training that is 16% faster than RLFN's two-stage process.

C. Benchmarking Protocol

• Evaluated 11 real-time SR models (including ESPCN, RLFN, SPAN, XLSR, and NVIDIA VSR).
• Conducted both pre-trained and fine-tuned evaluations on the StreamSR dataset.
• Used 7 objective metrics (PSNR, SSIM, LPIPS, MUSIQ, CLIP-IQA, etc.) and a large-scale subjective user study involving 3,822 participants and 37,184 pairwise comparisons (Bradley-Terry model).

3. Key Contributions

1. StreamSR Dataset: A comprehensive, real-world streaming benchmark containing 5,200 compressed video triplets, addressing the lack of compression-aware data in current SR research.
2. EfRLFN Model: A new SoTA real-time SR model that achieves an optimal trade-off between quality and speed. Key innovations include the use of tanh activation and ECA, which improve feature preservation and efficiency.
3. Comprehensive Benchmark: The first systematic evaluation of 11 real-time models on compressed streaming content, revealing that fine-tuning on real-world data significantly boosts performance even for models not originally designed for it.
4. Open Resources: The authors released the dataset, code, pre-trained weights, and raw subjective data (37k+ responses) to foster reproducibility.

4. Results

• Performance on StreamSR: EfRLFN achieved the highest subjective scores (3.33) among real-time models, outperforming NVIDIA VSR (2.57) and SPAN (2.55). It was preferred by users in 77.4% of comparisons against NVIDIA VSR.
• Objective Metrics: EfRLFN set new records in PSNR (37.85), SSIM (0.928), and LPIPS (0.059) on the 2× track, while maintaining 271 FPS on an RTX 2080 (well above the 30 FPS real-time threshold).
• Generalization: Fine-tuning other models (e.g., SPAN, RLFN) on StreamSR resulted in significant gains, proving the dataset's utility as a general training resource.
• Efficiency: EfRLFN is 15% faster than RLFN and achieves lower latency on TensorRT (12.07ms for 2× upscaling) compared to RLFN (29.12ms).
• Cross-Dataset Validation: EfRLFN consistently outperformed competitors on standard benchmarks (Set14, BSD100, Urban100, DIV2K) even without fine-tuning, demonstrating robust generalization.

5. Significance

This paper fundamentally shifts the paradigm for real-time super-resolution research by highlighting that training data quality is as critical as model architecture.

• Practical Impact: It provides a viable solution for streaming services to enhance video quality without increasing bandwidth, directly addressing the "compression vs. quality" trade-off.
• Research Direction: The study demonstrates that lightweight, image-based models can outperform complex video-based models in real-time scenarios if trained on appropriate data.
• Community Resource: By releasing a massive, high-quality dataset and subjective preference data, the authors enable future research into better quality assessment metrics and more efficient SR architectures.

In conclusion, the work establishes EfRLFN as the new benchmark for real-time streaming SR and provides the StreamSR dataset as the essential foundation for future advancements in compression-aware video enhancement.