ContrastiveCFG: Guiding Diffusion Sampling by Contrasting Positive and Negative Concepts

Technical Summary: ContrastiveCFG

Problem Statement
Classifier-Free Guidance (CFG) has become the standard for improving condition alignment in diffusion model sampling by sharpening the posterior distribution. However, a common application of CFG is "Negative Prompting" (NP), where users attempt to filter out unwanted features by simply negating the CFG guidance vector (i.e., $p(x)/p(c|x)^\gamma$). The paper identifies a critical flaw in this naive approach: negating the guidance creates an inverted probability distribution. As the guidance scale $\gamma$ increases, this inverted distribution dominates the marginal data distribution, causing samples to drift into low-density regions or completely outside the original data support. This results in significant degradation of sample quality and artifacts. While recent methods like Dynamic Negative Guidance (DNG) attempt to mitigate this by using complementary conditions, they struggle when positive and negative conditions are not mutually exclusive (e.g., overlapping text prompts), often failing to fully remove unwanted concepts without compromising the desired ones.

Methodology
The authors propose ContrastiveCFG (CCFG), a novel guidance strategy that reformulates conditional sampling as an optimization problem using contrastive loss, specifically derived from Noise Contrastive Estimation (NCE).

1. Core Concept: Instead of inverting the probability distribution, CCFG treats guidance as an objective to either attract the denoising direction toward a positive condition ($c_+$) or repel it from a negative condition ($c_-$) using a contrastive loss function.
2. Derivation:
   • The method parameterizes the data distribution $p_\theta(x)$ using a pre-trained diffusion model and defines a noise distribution $q(x)$ based on the unconditional (or alternative) prediction.
   • For positive guidance, the model minimizes a loss that encourages the predicted noise $\epsilon$ to align with the conditional noise $\hat{\epsilon}_{c}$ while distinguishing it from the unconditional noise $\hat{\epsilon}_{\emptyset}$.
   • For negative guidance, the roles are reversed: the model minimizes a loss that pushes the predicted noise away from the unwanted condition $\hat{\epsilon}_{c_-}$ while keeping it close to the unconditional baseline.
3. Sampling Strategy:
   • The authors derive a closed-form update rule for the denoising direction $\hat{\epsilon}$ based on the gradient of the contrastive loss.
   • The resulting guidance term includes a dynamic weighting coefficient (dependent on the distance between the conditional and unconditional predictions).
   • Crucially, as the current sample becomes increasingly unrelated to the negative condition, the negative guidance strength automatically attenuates to zero. This prevents the "over-repelling" and distributional shifts seen in standard NP.
   • The method integrates seamlessly into existing deterministic DDIM sampling with minimal computational overhead, requiring only additional forward passes for the conditional and unconditional models.

Key Contributions

• Novel Guidance Formulation: The paper introduces a contrastive loss-based framework for diffusion guidance that avoids the mathematical pitfalls of inverting probability distributions inherent in standard Negative Prompting.
• Dynamic Attenuation: Unlike static negation methods, CCFG automatically controls the strength of repulsion. The guidance force diminishes as the sample moves away from the unwanted concept, ensuring the sampling process remains within the valid data manifold.
• Unified Framework: The approach handles both positive and negative conditions within a single, consistent mathematical formulation, allowing for simultaneous attraction to desired concepts and repulsion from undesired ones without requiring complex empirical adjustments or hyperparameter tuning for conflict resolution.

Experimental Results
The authors evaluated CCFG across diverse scenarios, including class-conditional generation (MNIST, CIFAR10, ImageNet-1k) and text-to-image (T2I) generation (StableDiffusion 1.5, SDXL).

• Class-Conditional Generation: On ImageNet-1k, CCFG achieved the highest "Overall Accuracy" (simultaneous positive alignment and negative exclusion) compared to baselines like CFG++, NP, DNG, Perp-Neg, and SLD. It successfully removed unwanted classes (e.g., distinguishing similar breeds like Maltese vs. Siberian Husky) while maintaining high aesthetic scores and image quality, whereas NP degraded image quality and DNG failed to fully exclude overlapping classes.
• Text-to-Image Generation: In T2I tasks, CCFG demonstrated superior performance in removing unwanted concepts from complex, overlapping text prompts without distorting the positive prompt or introducing artifacts. Visual examples showed that while NP often removed entire categories (e.g., removing all red fruits when asked to remove "strawberries") or degraded image coherence, CCFG precisely targeted the unwanted features.
• Quantitative Benchmarks: On the proposed PIE-Bench-Neg dataset, CCFG achieved state-of-the-art performance in both positive prompt alignment (measured by CLIP, ImageReward, HPS-v2) and negative prompt suppression, outperforming other advanced negative guidance methods.

Significance and Claims
The paper claims that ContrastiveCFG offers a theoretically sound and practically effective solution to the limitations of existing negative guidance methods. By replacing the naive inversion of probabilities with a contrastive optimization objective, CCFG achieves a similar guiding effect to traditional CFG for positive conditions while overcoming the sample quality degradation and distributional shifts associated with Negative Prompting. The authors emphasize that their method faithfully removes unwanted concepts without sacrificing sample quality, even in scenarios where positive and negative conditions overlap, making it a robust tool for fine-grained control in diffusion models.