Training-Free Debiasing of Diffusion Models via CLIP-Guided Denoising Optimization

Technical Summary: Training-Free Debiasing of Diffusion Models via CLIP-Guided Denoising Optimization

Problem Statement
Text-to-image (T2I) diffusion models, such as Stable Diffusion, generate high-quality images but often exhibit significant demographic bias. Despite receiving neutral prompts (e.g., "a photo of a CEO"), these models consistently produce stereotypical representations regarding gender and race, reinforcing existing social biases and leading to the systematic underrepresentation of certain groups. Existing solutions face critical limitations: training-based approaches (retraining or fine-tuning) are computationally expensive and difficult to apply to deployed or closed-source models, while inference-time methods often rely on heuristic prompt engineering or static latent-space manipulations that degrade image quality, semantic alignment, or lack dynamic adaptability.

Methodology: Text Embedding Steering (TES)
The authors propose Text Embedding Steering (TES), a training-free framework that mitigates demographic bias by directly optimizing conditional text embeddings during the diffusion process without modifying model parameters. The core observation driving TES is that demographic attributes are largely determined in the early stages of denoising when the global image structure forms.

The method operates through a two-stage optimization strategy integrated into the standard classifier-free guidance (CFG) pipeline using DDIM sampling:

1. Early-Stage Global Alignment: Before fine-grained visual details emerge, TES performs an initial gradient update on the text embedding to steer the global generation trajectory toward a predefined target demographic distribution.
2. Iterative Denoising-Time Refinement: Following the initial alignment, TES continuously refines the embedding throughout subsequent denoising steps. This refinement utilizes CLIP-based feedback derived from intermediate "clean-image" estimates ($\hat{x}_0$) reconstructed at each timestep.

Optimization Objective
TES optimizes an additive perturbation ($\Delta e$) to the original text embedding ($e_0$), such that $e_t = e_0 + \Delta e$. The optimization minimizes a composite loss function ($L_{total}$) computed over a specific timestep window where the reconstructed image is semantically meaningful but the trajectory remains flexible:

• Attribute Loss ($L_{attr}$): Encourages the generated image to align with a target attribute description (e.g., "a female person") while opposing a contrasting attribute (e.g., "a male person"), using CLIP cosine similarity.
• Profession Loss ($L_{prof}$): Preserves the semantic content of the original prompt (e.g., "CEO") to prevent the model from drifting away from the intended subject.
• Regularization ($L_{reg}$): An $L_2$ penalty on the embedding perturbation to prevent excessive deviation from the original embedding.

The framework dynamically adjusts the learning rate based on the timestep, applying smaller updates during high-noise phases and increasing strength as reliable semantic information becomes available.

Key Contributions

1. Training-Free Framework: TES mitigates demographic bias by continuously optimizing text embeddings during inference, requiring no model retraining, fine-tuning, or auxiliary modules.
2. Two-Stage Strategy: The authors identify the critical importance of early-stage intervention for global trajectory control, combined with iterative refinement to maintain stability and prevent drift back to biased outcomes.
3. Dynamic Context-Aware Control: Unlike methods relying on fixed directions or static mappings, TES adapts the embedding based on intermediate clean-image estimates, enabling context-aware demographic steering.

Experimental Results
Extensive experiments were conducted on Stable Diffusion v2.1 (and v1.5 for generalization) across six profession categories (e.g., CEO, nurse, taxi driver) and two demographic attributes (gender and race).

• Fairness: TES significantly outperforms existing training-free baselines (e.g., LightFair, EntiGen) and the original Stable Diffusion. For gender debiasing, it reduced the Bias-O metric from 0.85 (baseline) to 0.26 and the Fairness Discrepancy (FD) from 0.54 to 0.13. For race debiasing, Bias-O dropped from 0.63 to 0.15.
• Quality and Semantics: Despite aggressive debiasing, TES maintains competitive image quality. It achieved the best FID scores (226.13 for gender, 228.52 for race) and high CLIP-T scores, indicating strong text-image semantic alignment. The method successfully preserves profession semantics and visual coherence while altering demographic attributes.
• Ablation Studies: Results confirm that both the early-stage alignment and iterative refinement are necessary; using only one stage leads to either poor fairness control or unstable semantic alignment. Furthermore, the inclusion of the profession-preserving loss is essential for maintaining the intended subject matter.

Significance and Claims
The paper claims that TES demonstrates that inference-time text embedding optimization is a practical and scalable solution for fairness-aware generation. By decoupling bias mitigation from model retraining, the approach offers a plug-and-play mechanism that can be applied to existing, deployed diffusion models. The authors assert that their method achieves a superior balance between fairness and image quality compared to prior art, producing more balanced and realistic representations of social roles without sacrificing the reliability or visual fidelity of the generated content. The work highlights that dynamic, context-aware steering of embeddings is a viable path toward reducing systematic demographic biases in generative AI.