🤖 machine learning

GAN-Diff : Coupling Pretrained WGAN-GP Features with Conditional Diffusion U-Nets

This paper proposes GAN-Diff, a hybrid framework that leverages intermediate features from a frozen pretrained WGAN-GP generator as a prior to guide a conditional diffusion U-Net via cross-attention, achieving significant improvements in image restoration tasks like denoising and super-resolution while addressing key training instabilities.

Original authors: Saif Ahmed, Ashadulla Hil Galib, S. M. Riaz Rahman Antu, Ahmed Faizul Haque Dhrubo, Souvik Pramanik, Mohammad Abdul Qayum, Mohsin Sajjad, Mohammad Ashrafuzzaman Khan

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC0 1.0

Original authors: Saif Ahmed, Ashadulla Hil Galib, S. M. Riaz Rahman Antu, Ahmed Faizul Haque Dhrubo, Souvik Pramanik, Mohammad Abdul Qayum, Mohsin Sajjad, Mohammad Ashrafuzzaman Khan

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of digital imaging, computers have learned to create pictures from nothing, a feat that once seemed impossible. Two different families of artificial intelligence have emerged as the leaders in this field. The first, known as generative adversarial networks, works like a fast artist who can sketch a complete face in a single, swift motion. These systems are efficient but can sometimes struggle to stay consistent, occasionally producing images that look strange or unstable. The second family, called diffusion models, works more like a sculptor slowly chiseling a statue out of a block of stone. It starts with a chaotic cloud of static noise and gradually removes the confusion step by step until a clear image emerges. While this second method produces incredibly high-quality and detailed results, it is slow and computationally expensive, requiring many repetitive steps to finish a single picture. The challenge for scientists has been to combine the speed of the first approach with the precision of the second, creating a system that is both fast and reliable.

A team of researchers at North South University in Bangladesh has taken a significant step toward solving this puzzle. They developed a new hybrid framework that guides the slow, careful sculpting process of a diffusion model using the structural knowledge of a pre-trained, fast generator. In their work, they took a generator that had already been trained to create realistic human faces and froze it, meaning its internal settings were locked so it could not change. Instead of letting this frozen generator create the final image, they used it as a map. As the diffusion model worked to clean up a noisy or blurry photo, it looked at the intermediate features from the frozen generator to understand what the underlying structure of a face should look like. This guidance was provided through a mechanism that allowed the cleaning model to pay attention to specific parts of the generator's map, ensuring that the eyes, nose, and mouth remained in the correct places while the noise was removed.

The researchers tested this system on two specific tasks: removing grainy noise from face images and doubling the resolution of low-quality pictures. They used a dataset of fifty thousand celebrity faces, focusing their evaluation on five specific individuals to ensure they could track changes accurately from start to finish. Before their final system worked, they had to overcome several hurdles that caused the training process to become unstable. They found that if the two parts of the initial generator learned at different speeds, the system would fail. They also discovered that starting the cleaning process with the wrong kind of noise or using too much corruption made the results poor. By carefully adjusting these factors, including how often the system averaged its own settings to smooth out errors, they created a stable pipeline.

The results showed a clear improvement in image quality. For the task of removing noise, the new method improved the clarity of the images by 4.40 decibels compared to the original noisy input. For the task of making low-resolution images sharper, it improved the quality by 3.70 decibels over a standard baseline method. These numbers represent a measurable gain in how closely the restored image matches the original, clean face. Visually, the system successfully removed visible speckles and retained important identity features like the shape of the eyes and the contours of the hair. In the super-resolution task, it was able to synthesize realistic details, such as fine strands of hair and skin texture, that were missing from the low-resolution inputs, without creating the blocky artifacts often seen in other methods.

A key insight from their work was how the guidance from the frozen generator should be applied. They found that the system worked best when the structural map from the generator was calculated once at the beginning of the process and then held constant throughout the entire cleaning sequence. If the system tried to recalculate this map at every single step, the instructions became contradictory, confusing the model. By keeping the guidance steady, the diffusion model could follow a consistent path to a clear result. The researchers noted that while their method worked well on the five tracked faces, the evaluation was limited to this small group, and they did not test how the system would handle different levels of noise or degradation beyond what they specifically set. They also did not include certain perceptual metrics that measure how human-like the images appear, relying instead on standard mathematical measures of clarity and structure.

This work demonstrates that a frozen generator can serve as a reliable guide for a diffusion model, helping it to restore images more effectively. The approach suggests that combining the structural strengths of one type of AI with the iterative refinement of another can lead to better outcomes than using either alone. The researchers identified specific causes for instability in such hybrid systems and provided solutions that allowed their model to function smoothly. While the study is limited to face images and specific types of degradation, the findings offer a clear path forward for building more robust image restoration tools that can handle the complex task of turning a damaged picture back into a clear one.

Technical Summary: GAN-Diff – Coupling Pretrained WGAN-GP Features with Conditional Diffusion U-Nets

Problem Statement

Deep generative models have become essential for image synthesis and recovery, yet distinct architectures possess complementary strengths and weaknesses. Generative Adversarial Networks (GANs), particularly those utilizing Wasserstein distance with gradient penalty (WGAN-GP), offer efficient, single-pass inference and stable training but can suffer from mode collapse and training instability. Conversely, Diffusion Probabilistic Models (DDPMs) provide high-quality image restoration and diversity through iterative denoising but require significant computational resources due to their sequential sampling nature.

The core challenge addressed by this work is the integration of these two paradigms to leverage the structural modeling capabilities of GANs to guide the restoration process of diffusion models without compromising the latter's denoising abilities. Specifically, the authors aim to determine if a frozen, pretrained WGAN-GP generator can serve as an effective feature-level prior to steer a conditional diffusion U-Net, thereby improving performance in Gaussian denoising and 2× super-resolution tasks while mitigating common training instabilities.

Methodology

The proposed framework, GAN-Diff, employs a sequential multi-stage pipeline that couples a frozen WGAN-GP generator with a conditional diffusion U-Net.

1. Phase 1: WGAN-GP Prior Generator

A WGAN-GP generator ( $G_\theta$ ) is pretrained on the CelebA dataset (50,000 images, 64×64 resolution). The generator maps a 128-dimensional latent vector to a 3-channel image using five transposed-convolution blocks. A critic ( $C_\theta$ ) is trained simultaneously using the WGAN-GP objective with a gradient penalty ( $\lambda_{gp}=10$ ). Crucially, an Exponential Moving Average (EMA) copy of the generator is maintained to reduce parameter variance. Once training is complete, the generator is frozen to serve as a fixed feature extractor.

2. Phase 2: GAN-Guided Conditional Diffusion

The core innovation lies in how the frozen GAN features are integrated into the diffusion process:

Feature Extraction: For each sampling chain, a latent vector $z$ is passed through the frozen generator to extract an intermediate feature map $g$ (specifically at the 8×8 spatial resolution).
Cross-Attention Mechanism: This feature map $g$ is treated as a fixed context. It is used as the Key ( $K$ ) and Value ( $V$ ) in cross-attention blocks within the diffusion U-Net, while the noisy image features serve as the Query ( $Q$ ).
Fixed Prior: Unlike previous approaches that might resample GAN features at every step, this framework calculates $g$ once per sampling chain and keeps it constant throughout the entire DDIM (Denoising Diffusion Implicit Models) reverse process. This ensures trajectory consistency.
Tasks: The framework is adapted for two low-level vision tasks:
- Gaussian Denoising: Starting from a corrupted input ( $\sigma=0.15$ ).
- Super-Resolution: Starting from a bicubic downsampled/upsampled input (2× factor).

3. Addressing Instabilities

The development process identified and resolved four specific sources of instability:

Learning Rate Imbalance: Corrected the generator/critic learning rate ratio to the standard symmetric $2 \times 10^{-4}$ (avoiding the 4x imbalance that caused critic saturation).
DDIM Initialization: Adjusted initialization strategies for denoising versus super-resolution to prevent poor convergence.
Corruption Intensity: Calibrated the severity of input corruption to ensure the model learns robust features without being overwhelmed.
EMA Smoothing: Implemented appropriate EMA decay (0.9999) to stabilize parameter updates.

Key Contributions

Hybrid Architecture: A novel conditional diffusion network where a frozen WGAN-GP generator provides intermediate spatial features via cross-attention, acting as a structural prior for the diffusion U-Net.
Fixed Prior Strategy: The demonstration that calculating the GAN feature map once per sampling chain (rather than at every diffusion step) is critical for maintaining semantic consistency and trajectory stability in DDIM sampling.
Stability Diagnosis: A systematic identification and resolution of pipeline instabilities, specifically focusing on learning rate ratios, initialization protocols, and corruption normalization, which are often overlooked in hybrid GAN-diffusion research.
Empirical Validation: Successful application to Gaussian denoising and 2× super-resolution on the CelebA dataset, showing consistent improvements over standard baselines.

Results

The framework was evaluated on five fixed tracked faces from the CelebA dataset to ensure consistent quantitative comparison.

Denoising Performance: The GAN-guided diffusion model improved the Peak Signal-to-Noise Ratio (PSNR) by 4.40 dB (from 22.76 dB to 27.17 dB) and the Structural Similarity Index (SSIM) by 0.167 over the corrupted input baseline.
Super-Resolution Performance: The model improved PSNR by 3.70 dB (from 27.24 dB to 30.94 dB) and SSIM by 0.044 over the bicubic upsampling baseline.
Qualitative Analysis: Visual inspection confirmed the removal of speckle noise and the preservation of identity features (eyes, hair, mouth). In super-resolution, the model synthesized realistic high-frequency details (fine hair strands, skin texture) without introducing checkerboard artifacts, outperforming standard bicubic interpolation.
Training Dynamics: The WGAN-GP losses converged smoothly without oscillation, and the diffusion U-Nets showed steady monotonic loss decreases, validating the stability of the corrected pipeline.

Significance and Claims

The paper claims that its primary contribution is demonstrating the potential of a frozen GAN feature prior to guide diffusion models toward stable and effective image restoration. The authors emphasize that the integration of GANs and diffusion models should not merely substitute one for the other but should utilize the GAN's ability to model structural and semantic manifolds to inform the diffusion process.

The significance of this work lies in:

Efficiency and Quality: Achieving high-quality restoration that combines the structural guidance of GANs with the iterative refinement of diffusion models.
Robustness: Providing a "diagnosis-driven" solution to common instabilities in hybrid models, offering a reproducible path for future research in this domain.
Methodological Insight: Establishing that the GAN conditioning signal should be treated as a constant input throughout the DDIM sampling process to avoid contradictory semantic instructions, a finding that improves the reliability of the generation trajectory.

The authors maintain a modest stance, acknowledging limitations such as the small evaluation set (five faces), the lack of perceptual metrics (LPIPS/FID) in the final results, and the testing of only a single noise level. They suggest future work should involve larger test sets, perceptual metric evaluation, and ablation studies on the cross-attention conditioning.