Technical Summary: Direct Diffusion Score Preference Optimization (DDSPO)

Problem Statement

Diffusion models have achieved significant success in generative tasks, particularly text-to-image synthesis. However, they frequently struggle to align generated outputs with nuanced user intent and to consistently maintain high aesthetic quality. Existing preference-based training methods, such as Diffusion Direct Preference Optimization (Diffusion DPO), attempt to address these issues by learning from pairwise preferences over final samples.

A critical limitation of current approaches like Diffusion DPO is their reliance on approximating backward denoising transitions using the forward process $q(x_{t-1} | x_t, x_0)$ derived from terminal samples ($x_0$). This approximation does not reflect the model's actual reverse denoising process at inference time. Consequently, the supervision signal provided is misaligned with the model's true denoising trajectories, failing to offer effective stepwise supervision.

Methodology

The authors propose Direct Diffusion Score Preference Optimization (DDSPO), a framework that formulates preference supervision directly in the score space at each diffusion timestep, rather than in the final sample space.

Core Concept: Stepwise Contrastive Policy-Pair Supervision

Unlike prior methods that define preferences over final outputs $(x_0^w, x_0^l)$, DDSPO introduces stepwise supervision by comparing the denoising scores of a contrastive policy pair at each timestep $t$. The preference labels are extended to tuples of denoising transitions $((x_t^w, x_{t-1}^w), (x_t^l, x_{t-1}^l))$, derived from preferred and dispreferred denoising policies.

The objective function optimizes the model's denoising predictions toward targets $\boldsymbol{\epsilon}_\star^w$ and $\boldsymbol{\epsilon}_\star^l$, corresponding to preferred and dispreferred behaviors at each step. The loss function is defined as:
$$\mathcal{L}_{\text{DDSPO}}(\theta) = - \mathbb{E} \left[ \log \sigma \left( - \beta \cdot \left[ \| \boldsymbol{\epsilon}_\star^w - \boldsymbol{\epsilon}_\theta(x_t^w, t, c) \|_2^2 - \| \boldsymbol{\epsilon}_\star^w - \boldsymbol{\epsilon}_{\text{ref}}(x_t^w, t, c) \|_2^2 - (\| \boldsymbol{\epsilon}_\star^l - \boldsymbol{\epsilon}_\theta(x_t^l, t, c) \|_2^2 - \| \boldsymbol{\epsilon}_\star^l - \boldsymbol{\epsilon}_{\text{ref}}(x_t^l, t, c) \|_2^2) \right] \right) \right]$$
Here, $\boldsymbol{\epsilon}_\star^w$ and $\boldsymbol{\epsilon}_\star^l$ are denoising score targets obtained from the contrastive policy pair, retaining actual backward denoising transitions rather than forward-process approximations.

Instantiations of the Contrastive Policy Pair

The paper proposes two practical ways to instantiate the contrastive policy pair:

1. Data-Driven Contrastive Policy Pair (DD-CPP):

   • Requires a preference-labeled dataset of image pairs $(x_0^w, x_0^l, c)$.
   • Two separate models are fine-tuned from a pretrained initialization: a "winning" model $\phi^w$ on preferred samples and a "losing" model $\phi^l$ on dispreferred samples.
   • The supervision targets are the denoising scores predicted by these specialized models: $\boldsymbol{\epsilon}_\star^w \leftarrow \boldsymbol{\epsilon}_{\phi^w}(x_t, t, c)$ and $\boldsymbol{\epsilon}_\star^l \leftarrow \boldsymbol{\epsilon}_{\phi^l}(x_t, t, c)$.

2. Training-Free Contrastive Policy Pair (TF-CPP):

   • Requires no additional training or preference-labeled data.
   • Utilizes a single pretrained reference model.
   • Constructs a contrastive pair by using the original prompt $c$ (preferred) and a semantically degraded variant $c^-$ (dispreferred).
   • The preferred target is $\boldsymbol{\epsilon}_{\text{ref}}(x_t, t, c)$, and the dispreferred target is $\boldsymbol{\epsilon}_{\text{ref}}(x_t, t, c^-)$.
   • This approach leverages the observation that at larger timesteps, the reverse update is largely driven by the conditioning signal, allowing $c^-$ to provide a meaningful negative score target even without generating a specific dispreferred image $x_0^l$.

Key Contributions

1. DDSPO Framework: A preference optimization framework enabling direct supervision over intermediate denoising steps in score space via a contrastive policy pair, avoiding reliance on forward-process approximations.
2. Practical Instantiations: The introduction of DD-CPP (learning policies from labeled data) and TF-CPP (constructing contrastive directions from a reference model using prompt degradation), the latter requiring neither reward modeling nor manual annotations.
3. Empirical Validation: Extensive experiments demonstrating that stepwise supervision derived from contrastive policy pairs is more effective than forward-process-based supervision for both text–image alignment and aesthetic quality tasks.

Experimental Results

The authors evaluated DDSPO on text-to-image alignment and aesthetic quality improvement tasks using Stable Diffusion variants (SD-1.4, SD-1.5, SDXL) and DiT-based models (SANA, SD3-M).

• Text-to-Image Alignment:

  • Using TF-CPP (without human-annotated data), DDSPO achieved state-of-the-art results on GenEval and T2I-CompBench benchmarks compared to Diffusion DPO, DSPO, and other baselines.
  • DDSPO outperformed SOTA methods like Itercomp and CaPO on CompBench without requiring human-annotated preference data or reward models.
  • Qualitative results showed clearer separation between conditional distributions and better adherence to complex prompts compared to Diffusion DPO and DSPO.

• Aesthetic Quality:

  • Using DD-CPP on the Pick-a-Pic dataset, DDSPO improved HPSv2 and PickScore metrics over SD-1.5 and SDXL, outperforming Diffusion DPO and Diffusion KTO.
  • The TF-CPP variant (without human data) remained competitive with supervised methods.

• Ablation Studies:

  • Different prompt degradation strategies (e.g., random token removal vs. LLM-based rewriting) were tested. Random removal performed best for alignment, while LLM-based degradation tailored to aesthetics performed best for aesthetic quality.
  • An efficient variant of TF-CPP that reuses preferred images for negative targets (avoiding explicit generation of dispreferred images) showed performance comparable to the full setup, highlighting the flexibility of the framework.

Significance and Claims

The paper claims that the primary significance of DDSPO lies in its ability to provide stepwise supervision aligned with the model's actual backward denoising process. By defining preferences directly over local backward transitions ($x_t \to x_{t-1}$) rather than relying on forward-process surrogates, DDSPO offers richer, transition-level supervision.

The authors assert that this approach leads to more robust learning signals across the denoising trajectory, resulting in consistent improvements in both semantic alignment and aesthetic quality. Furthermore, the Training-Free Contrastive Policy Pair (TF-CPP) demonstrates that high-quality preference optimization can be achieved without the cost of collecting human preference annotations or training reward models, making the method highly scalable and data-efficient. The work positions DDSPO as a generalizable framework applicable to both diffusion and flow-based generative models.