Unconditional Priors Matter! Improving Conditional Generation of Fine-Tuned Diffusion Models

Technical Summary: Unconditional Priors Matter!

Title: Unconditional Priors Matter! Improving Conditional Generation of Fine-Tuned Diffusion Models
Authors: Prin Phunyaphibarn, Phillip Y. Lee, Jaihoon Kim, Minhyuk Sung (KAIST)

---

1. The Problem: Degradation of Unconditional Priors

The paper addresses a critical, often overlooked issue in Classifier-Free Guidance (CFG)-based conditional diffusion models.

In standard practice, a single network is fine-tuned to predict both conditional noise ($\epsilon_\theta(x_t, c)$) and unconditional noise ($\epsilon_\theta(x_t, \emptyset)$). This is typically achieved by randomly dropping the conditioning signal (e.g., 5–20% of the time) during training. The authors observe two major flaws in this approach:

• Capacity Bottleneck: Because the same network must learn both complex conditional distributions and the even more complex unconditional distribution, the "bandwidth" for the unconditional case is limited.
• Prior Degradation: Fine-tuning a base model (like Stable Diffusion) on a specific, narrower task (like novel view synthesis or image editing) causes the model to lose its rich, general-purpose unconditional knowledge.

Consequence: Since CFG relies on the difference between conditional and unconditional noise to "guide" the generation, a poor (low-quality/distorted) unconditional prior leads to significant degradation in the final conditional output, manifesting as color saturation, shape distortions, or loss of semantic detail.

2. Methodology: Unconditional Noise Replacement

The authors propose a training-free solution that leverages the high-quality unconditional priors of existing pretrained models.

The Core Idea: Instead of using the fine-tuned model's own (degraded) unconditional noise prediction, replace it with the noise prediction from a model that possesses a superior unconditional prior.

Mathematical Formulation:
In standard CFG, the guided noise is:
$$\epsilon_\theta^{(\gamma)}(x_t, c) = \epsilon_\theta(x_t, \emptyset) + \gamma(\epsilon_\theta(x_t, c) - \epsilon_\theta(x_t, \emptyset))$$

The authors propose the Modified CFG:
$$\epsilon_{\theta, \psi}^{(\gamma)}(x_t, c) = \epsilon_\psi(x_t, \emptyset) + \gamma(\epsilon_\theta(x_t, c) - \epsilon_\psi(x_t, \emptyset))$$
Where:

• $\epsilon_\theta$ is the noise from the fine-tuned conditional model.
• $\epsilon_\psi$ is the noise from a pretrained model with strong priors (which could be the original base model or even a completely different architecture like PixArt-$\alpha$).

Key Insight: The "replacement" model ($\psi$) does not need to share the same architecture or even the same training history as the fine-tuned model ($\theta$), as long as its unconditional distribution $p_\psi(x_t)$ is a better approximation of the true data distribution.

3. Key Contributions

• Identification of a Fundamental Flaw: They demonstrate that the degradation of unconditional priors during fine-tuning is a primary driver of poor conditional generation quality.
• Training-Free Solution: They provide a plug-and-play method that requires no retraining, no architectural changes, and no additional training data.
• Cross-Architecture Compatibility: They prove that the replacement model can have a different architecture (e.g., using a Diffusion Transformer/DiT to guide a UNet).
• Versatility: The method is shown to work across diverse modalities, including image-to-image, text-to-image, image-to-video, and novel view synthesis.

4. Experimental Results

The authors validated the method across five major conditional models:

• Zero-1-to-3 (Novel View Synthesis): Improved lighting, reduced color saturation, and corrected shape distortions. Quantitative metrics (LPIPS, PSNR, SSIM) showed significant gains, especially when using SD2.1 as the prior.
• Versatile Diffusion (Image Variations): Improved FID and FDDINOv2 scores while maintaining high condition alignment (CLIP-I). It effectively corrected excessive saturation and object distortions.
• DiT (Class-Conditional): Demonstrated that even for Transformer-based architectures, using the base model's prior improves FID, particularly when fine-tuning datasets are large.
• DynamiCrafter (Image-to-Video): Improved temporal consistency and aesthetic quality, resulting in videos with fewer distortions (e.g., hands and faces).
• InstructPix2Pix (Image Editing): Enhanced the faithfulness of edits (alignment with text instructions) while better preserving the identity of the original subject.

5. Significance and Limitations

Significance:
This work changes the perspective on CFG training. It suggests that we do not necessarily need to learn unconditional and conditional noise jointly. This could potentially allow for more efficient fine-tuning processes where the model focuses solely on learning the conditional mapping, relying on external models for the "unconditional" component.

Limitations:

• Memory/Speed: It requires loading a second model into VRAM, increasing memory usage. There is also a slight increase in inference time due to the extra forward pass required for the second model.
• Scope: The method is less effective for adapter-based methods like ControlNet, which do not suffer from the same level of unconditional prior degradation.