Technical Summary: DeDPO: Debiased Direct Preference Optimization for Diffusion Models

1. Problem Statement

Direct Preference Optimization (DPO) has become a standard method for aligning diffusion models with human preferences, offering an efficient alternative to Reinforcement Learning from Human Feedback (RLHF) by eliminating the need for a separate reward model. However, DPO faces a critical scalability bottleneck: it relies on large-scale, high-quality human preference labels, which are prohibitively expensive and time-consuming to acquire.

Recent attempts to address this by using synthetic preferences (generated by AI models) often fail because these synthetic labels can be noisy, inconsistent with true human judgment, or introduce systematic bias. Training directly on such imperfect feedback leads to unstable alignment and degraded performance. The core challenge is to leverage large volumes of inexpensive synthetic data while correcting for its inherent noise and bias, using only a small set of high-quality human labels.

2. Methodology: Debiased DPO (DeDPO)

The authors propose DeDPO, a semi-supervised framework that integrates a debiased estimation technique from causal inference into the DPO objective. The method operates under a setting where a small set of data is labeled with ground-truth human preferences, and a larger set is labeled with synthetic preferences generated by cost-effective AI feedback (e.g., Vision-Language Models or self-training).

2.1. Reformulating DPO as Binary Classification

The authors first re-interpret the standard Diffusion-DPO loss as a binary classification problem. For an image pair $(x_0, x_1)$ and prompt $c$, a preference label $z$ indicates which image is preferred. The DPO loss is rewritten as a binary cross-entropy loss $L(G_\theta(y), z)$, where $G_\theta$ is the model's implicit margin predictor.

2.2. The Debiased Loss Function

In a naive semi-supervised approach, one might simply impute missing human labels with synthetic labels $\hat{G}(y)$, leading to a biased Outcome Regression (OR) estimator. DeDPO instead proposes a debiased loss (Eq. 8) that combines the labeled and unlabeled data:

$$L_{DeDPO}(\theta) = \mathbb{E}_{n_l+n_u} [L(G_\theta(y), \hat{G}(y))] + \mathbb{E}_{n_l} [L(G_\theta(y_l), z_l) - L(G_\theta(y_l), \hat{G}(y_l))]$$

Where:

• $n_l$ and $n_u$ are the number of labeled and unlabeled samples.
• $z_l$ is the ground-truth human label.
• $\hat{G}(y)$ is the synthetic label.

Mechanism of Correction:

• Unlabeled Data: The loss uses the synthetic pseudo-labels $\hat{G}(y)$ directly as the supervision signal.
• Labeled Data: The loss introduces a correction term $(z_l - \hat{G}(y_l))$ that shifts the pseudo-label toward the true human label. This correction is amplified by an importance weight $\frac{n_l + n_u}{n_l}$, which reflects the scarcity of labeled data.
• Theoretical Properties:
  1. Unbiasedness: The expectation of the DeDPO loss equals the population-level DPO loss, regardless of whether the synthetic labels are correct. If synthetic labels are perfect, it utilizes all data; if they are completely wrong, it asymptotically reverts to using only the labeled data.
  2. Robustness to Synthetic Model Convergence: The authors prove that the target diffusion model $\theta$ can achieve optimal convergence rates even if the synthetic preference model $\hat{G}$ converges at a much slower rate (specifically, the error term depends on the 4th power of the synthetic model's error). This allows the system to be robust even when the synthetic annotator is imperfect.

2.3. Synthetic Preference Sources

The framework supports two primary sources for synthetic labels:

1. Self-Training: Using the implicit reward margin of the current policy $\theta$ (from previous iterations) to generate labels. The authors note that to avoid overfitting, the synthetic model should ideally be trained on data independent of the DPO training set.
2. Pretrained Vision-Language Models (VLMs): Using models like Qwen or CLIP to annotate image pairs. This is preferred in the paper's setting as it avoids the independence violation inherent in self-training on the same labeled data.

3. Key Contributions

• Debiased Estimation for Diffusion: The paper introduces the first application of debiased estimation techniques from causal inference to the problem of preference optimization in diffusion models.
• Semi-Supervised Alignment: It establishes a robust framework for aligning diffusion models using a small fraction of human labels (e.g., 25%) augmented by a large corpus of synthetic labels, without requiring a separate reward model.
• Theoretical Guarantees: The authors provide theoretical proofs demonstrating that the proposed loss is unbiased and that the learning process is robust to errors in the synthetic preference model, provided the synthetic model is trained on independent data.

4. Experimental Results

The authors evaluated DeDPO on the FiFA-5K and HPDv2 datasets using SD1.5 and SDXL backbones.

• Performance vs. Fully Supervised DPO: DeDPO trained with 25% human and 75% synthetic labels consistently matched or slightly exceeded the performance of DPO trained on 100% human labels.
  • On FiFA-5K with SD1.5, DeDPO achieved a PickScore of 21.91 and HPSv2 of 27.80, surpassing the fully supervised DPO baseline (21.88 PS, 27.79 HPSv2).
  • On SDXL, DeDPO matched the fully supervised performance (22.83 PS vs. 22.84 PS).
• Robustness to Noise: In contrast, a naive approach of mixing synthetic labels with DPO (without debiasing) consistently underperformed, showing that simply adding synthetic data can degrade alignment if not corrected.
• Scalability: As the size of the unlabeled synthetic dataset increased (up to 98K pairs), DeDPO maintained stable performance, whereas vanilla DPO performance degraded significantly due to the accumulation of noisy labels.
• Qualitative Improvements: Visual comparisons showed that DeDPO better captured fine-grained details and compositional constraints (e.g., specific objects in complex prompts) compared to models trained with fewer human labels or naive synthetic mixing.
• AI-as-a-Judge: Human-aligned AI judges (Gemini 2.5 Flash) preferred DeDPO over fully supervised DPO in ~57% of cases for General Preference and Visual Appeal, and over semi-supervised DPO in ~76% of cases.

5. Significance and Claims

The paper claims that DeDPO provides a scalable, cost-effective solution for human-AI alignment in diffusion models. By integrating debiased estimation, the method successfully transforms inexpensive, noisy synthetic feedback into a reliable training signal.

The authors assert that DeDPO:

1. Overcomes the data bottleneck: It enables high-quality alignment using a small fraction of human labels, making large-scale preference optimization feasible without prohibitive costs.
2. Ensures robustness: It is resilient to the variations and noise inherent in synthetic annotators (including self-training and VLMs), preventing the instability often seen when training on AI-generated preferences.
3. Achieves state-of-the-art efficiency: It achieves performance comparable to, and occasionally exceeding, models trained on fully human-labeled datasets, establishing a new standard for efficient preference optimization.

The work concludes that properly debiased synthetic feedback is a viable and reliable surrogate for extensive human labeling, offering a practical path forward for aligning generative models with human values.