Technical Summary: Fortifying Fully Convolutional Generative Adversarial Networks for Image Super-Resolution Using Divergence Measures

Problem Statement

Image Super-Resolution (SR) aims to reconstruct a High-Resolution (HR) image from a Low-Resolution (LR) input. This is an ill-posed problem because the initial LR transformation is typically non-invertible and lossy. While classical interpolation methods fail to generate commendable SR due to their reliance on local information, and deep Convolutional Neural Networks (CNNs) struggle with generalizability and distortion, Generative Adversarial Networks (GANs) have offered a breakthrough. However, existing GAN-based SR methods often suffer from the loss of fine details, the introduction of artifacts, and mode collapse, particularly when trained on smaller datasets. Furthermore, many approaches rely on pre-trained networks for perceptual loss, which may introduce bias, or fail to explicitly model the distributional relationship between LR and SR inputs.

Methodology: The SuRGe Architecture

The authors propose SuRGe (Super-Resolution Generator), a fully convolutional GAN-based architecture designed to address these limitations through three primary mechanisms: hierarchical feature mixing, novel divergence-based loss functions, and robust training strategies.

1. Generator Architecture ($G$)

Unlike standard approaches that perform 4x up-scaling in a single step at the end of the network, SuRGe employs a two-step up-scaling strategy (2x followed by 2x) to mitigate abrupt feature space distortions.

• Hierarchical Feature Extraction: The generator is divided into left and right halves. The left half extracts features through an initial convolution block ($C_0$) and a repetitive residual generator block ($R_0$).
• Adaptive Feature Mixing: To combine low-level features (from $C_0$) and high-level features (from $R_0$), SuRGe introduces learnable mixing modules ($F_0$ and $F_1$). These modules perform a convex combination of features from different network depths using learnable weights constrained by a Softmax activation. This allows the network to adaptively select the most useful features for reconstruction.
• Skip Connections: The architecture utilizes skip connections to propagate low-level features (e.g., colors, textures) through the network, ensuring minute details are recovered after up-scaling steps.
• Up-scaling Mechanism: Instead of using PixelShuffle or transposed convolutions, which can introduce checkerboard artifacts, SuRGe employs nearest-neighbor interpolation for up-scaling, followed by stabilization convolutions.

2. Discriminator Architecture ($D$)

The discriminator follows a structure similar to the generator to ensure close embeddings in the learned feature space but includes specific modifications:

• Repetitive Residual Blocks: It uses residual blocks with inter-sub-block skip connections.
• Activation and Normalization: It employs LeakyReLU to prevent sparse gradients and Pixel Normalization (instead of Batch Normalization) to avoid quality issues and adhere to WGAN-GP constraints.
• Progressive Kernel Sizes: The number of filters and kernel sizes increase gradually across blocks to balance low-level and high-level information capture and prevent overfitting.

3. Loss Functions

SuRGe introduces a novel combination of loss functions to guide the generator and discriminator:

• Generator Loss ($L_G$):
  • Adversarial Loss ($L_a$): Standard GAN loss to distinguish real HR from generated SR.
  • Jensen-Shannon (JS) Divergence ($L_{JS}$): Explicitly minimizes the dissimilarity between the distributions of the generated SR and the HR ground truth ($p_{G(x)}$ and $p_y$).
  • Gromov-Wasserstein (GW) Loss ($L_{GW}$): A divergence measure between metric spaces of potentially different dimensions. This is used to minimize the discrepancy between the LR input distribution ($p_x$) and the SR output distribution ($p_{G(x)}$), enforcing structural similarity even when dimensions differ.
  • Dynamic Weighting: Instead of static weights, the three loss components are combined using a dynamically calculated convex combination based on Softmax. This allows the model to adaptively adjust the focus of each loss during training.
• Discriminator Loss ($L_D$):
  • Wasserstein Loss with Gradient Penalty (WGAN-GP): Used to train the discriminator to prevent mode collapse and improve generation quality without the complexity of weight clipping.

Key Contributions

The paper claims the following primary contributions:

1. First Application of GW in GAN-SR: SuRGe is presented as the first super-resolution model to utilize the Gromov-Wasserstein (GW) divergence to fuel the learning of the generator, directly modeling the relationship between LR and SR distributions.
2. Divergence-Based Training: Moving away from pre-trained model-biased perceptual similarity, the generator is trained using JS divergence (between SR and HR) and GW divergence (between LR and SR), alongside the adversarial loss.
3. Adaptive Feature Mixing: The introduction of a generator that efficiently employs skip connections and learnable mixing modules ($F_0, F_1$) to garner and adaptively combine semantic information from different feature representation levels.
4. Robust Architecture: The use of nearest-neighbor up-scaling and specific normalization strategies to reduce artifacts and mode collapse.

Experimental Results

The efficacy of SuRGe was evaluated on 10 benchmark datasets against 28 state-of-the-art contenders (including CNN, GAN, and Transformer-based methods).

• Standard Benchmarks: On Set5, Set14, BSD100, and Urban100, SuRGe demonstrated superior performance. On average, it improved PSNR and SSIM by 4.26% and 5.72% respectively on the Set14, BSD100, and Urban100 datasets compared to the best existing methods.
• Complex Benchmarks: On six additional complex datasets (Kitti2012, Kitti2015, Middlebury, PIRM, OST300, MANGA109), SuRGe outperformed the state-of-the-art by an average of 17.59% in PSNR.
• Ablation Studies: Experiments confirmed that the progressive addition of components (ResNet-based discriminator, incremental kernel sizes, JS/GW losses, and dynamic weighting) leads to consistent performance gains.
• Qualitative Analysis: Visual comparisons show that SuRGe produces images with finer textures, better color fidelity, and more intricate details compared to SRGAN, BSRGAN, SWIN-IR, and LTE, while avoiding the excessive noise or distortion seen in ESRGAN variants.

Significance and Claims

The paper asserts that SuRGe offers a significant advancement in the field of image super-resolution by:

• Authentic SR Capabilities: The incorporation of the LR-SR relationship via GW divergence endows the model with "authentic super-resolution capabilities," moving beyond simple pixel mapping to distributional alignment.
• Efficiency: Despite its complex loss landscape and feature mixing, SuRGe maintains a low inference time and a smaller model size compared to typical GAN-based SR methods.
• Theoretical Novelty: It represents the first exploration of explicit divergence measures (specifically GW) in the context of GAN-based super-resolution, providing a new theoretical framework for aligning LR and SR distributions.
• Performance: The model achieves state-of-the-art results across diverse datasets, demonstrating that a fully convolutional approach with carefully designed divergence measures can outperform both traditional CNNs and complex Transformer-based architectures in terms of both quantitative metrics and visual quality.