Technical Summary: D2PO – Optimizing Diffusion Samplers via Dynamic Preference

1. Problem Statement

Diffusion Probabilistic Models (DPMs) have achieved state-of-the-art results in image, text-to-image, and video generation but suffer from high computational costs due to the iterative nature of their sampling process, which requires many function evaluations (NFE). While various acceleration methods exist, including numerical solvers and knowledge distillation, a fundamental limitation persists in student-teacher regression frameworks.

In these frameworks, a low-NFE "student" sampler is trained to mimic the trajectories or outputs of a fixed, high-NFE "teacher" using regression losses (e.g., $\ell_2$ or LPIPS). The paper identifies a critical structural bottleneck: as the gap between the student's NFE and the teacher's NFE widens (a common requirement for aggressive acceleration), the regression objective forces the student to prioritize low-level alignment with the teacher's coarse trajectory. This results in the suppression of high-frequency textures and fine-grained details, leading to a degradation in perceptual quality even when the teacher model is high-fidelity. Existing methods fail to leverage the full potential of high-quality teachers under low-NFE constraints because they are bound by a static error floor defined by the teacher's discretization.

2. Methodology: D2PO

To address this, the authors propose D2PO (Dynamic Direct Preference Optimization), a framework that reframes sampler optimization from a regression task to a preference-based alignment problem.

2.1 Energy-Based Surrogate for Deterministic Policies

A primary challenge in applying Direct Preference Optimization (DPO) to diffusion samplers is that the sampling policy is deterministic (induced by a fixed numerical solver), making the likelihood of the output degenerate and non-differentiable. D2PO overcomes this by modeling the policy-induced distribution as an Energy-Based Model (EBM).

• The policy $\pi_\phi$ is defined as a smooth surrogate where the energy $E(\mathbf{x})$ is the distance between a candidate sample $\mathbf{x}$ and the deterministic sampler output $\mathbf{x}_\phi$.
• This formulation allows the log-probability ratio required for DPO to be expressed as a difference in energy values, rendering the optimization tractable.

2.2 Novel Score-Based Distance Metric

Instead of relying on standard perceptual metrics like LPIPS or $\ell_2$, which often fail to capture high-frequency texture errors, D2PO introduces a score-based energy metric derived directly from the pretrained diffusion score network $s_\theta$.

• The energy is defined as the discrepancy between the score predictions of a candidate sample and the sampler output across multiple noise levels.
• Specifically, the distance $d_\theta$ measures the squared $\ell_2$ difference between noise predictions ($\epsilon_\theta$) at various timesteps.
• By integrating these discrepancies over the noise schedule with a specific weighting ($w(t) = \sigma_t^2$), the metric captures both coarse structural consistency (at high noise levels) and fine-grained textural details (at low noise levels), providing a richer preference signal than conventional metrics.

2.3 Dynamic Preference Mechanism

D2PO replaces the static teacher supervision with a dynamic reference mechanism that evolves alongside the student policy.

• Student Sampler ($\phi$): The target policy being optimized (parameters: timestep schedule $S$ and CFG weights $\omega$).
• Losing Sample ($x_l$): Generated by the student using a sparse, fast schedule, then degraded (e.g., via low-pass filtering) to simulate a lower-quality output.
• Winning Sample ($x_w$): Generated by a dynamic teacher ($\phi'$) derived from the current student parameters. The dynamic teacher uses a refined, denser timestep schedule (e.g., $2N$ steps vs. the student's $N$ steps) created by interpolating the student's current schedule.
• Reference Policy ($\phi_{ref}$): A policy that tracks the student's history (updated via EMA for CFG weights and epoch-wise copying for timesteps) to anchor the optimization.

This dynamic loop encourages the student to align with its own higher-fidelity, denser-schedule approximation. Theoretically, this minimizes the student's own discretization error rather than converging to a fixed residual error floor imposed by an external teacher.

3. Key Contributions

1. D2PO Framework: A preference-based optimization framework for diffusion samplers that establishes a tractable alignment objective by modeling the deterministic policy as an energy-based surrogate.
2. Score-Based Energy Metric: A novel distance metric derived from the pretrained score network that captures multi-scale structural and textural discrepancies, outperforming conventional perceptual losses in detecting discretization errors.
3. Dynamic Preference Loop: A self-improving mechanism where the "teacher" is a dynamically refined version of the current student policy. This eliminates the fixed error floor of static distillation and drives the sampler to minimize its own discretization error.
4. Comprehensive Validation: Extensive experiments demonstrating that D2PO learns superior sampling policies, outperforming state-of-the-art distillation-based baselines (e.g., LD3, GITS, DMN) across various solvers and datasets.

4. Experimental Results

The authors evaluated D2PO on text-to-image synthesis using Stable Diffusion v1.5, ImageNet-256 (latent space), and the InstaFlow model.

• Quantitative Performance:
  • Low NFE (4-5 steps): D2PO achieves state-of-the-art perceptual quality (HPSv2 and Aesthetic scores) and significantly lower FID scores compared to baselines. It effectively corrects foundational flaws like blurriness and artifacts that regression-based methods introduce.
  • High NFE (6-7 steps): D2PO maintains a significant lead in perceptual scores. While FID scores become comparable to baselines (reflecting the quality-diversity trade-off), D2PO's optimization toward perceptual quality remains superior.
  • Generalization: The method generalizes to different ODE solvers (iPNDM, UniPC, DPM-Solver++) and modern backbones (Stable Diffusion 3.5-Medium), consistently outperforming baselines.
• Ablation Studies:
  • Removing the dynamic preference (reverting to a fixed teacher) degrades performance, confirming the necessity of the self-improving loop.
  • Replacing the score-based energy with LPIPS causes a significant drop in fidelity, validating the metric's ability to capture fine-grained errors.
• Human Evaluation: In a blind user study, D2PO received 60.3% of votes for overall preference, far surpassing LD3 (27.6%) and GITS (12.1%), confirming that the metric improvements translate to human-perceived quality.
• Semantic Alignment: D2PO also achieves higher CLIP scores than baselines, suggesting that optimizing for preference alignment preserves prompt relevance better than rigid pointwise regression.

5. Significance and Claims

The paper claims that D2PO addresses a critical performance bottleneck in diffusion sampler optimization. By shifting from a regression-to-teacher paradigm to a dynamic preference alignment paradigm, D2PO unlocks the full potential of high-quality teachers without being constrained by their static discretization errors.

The authors assert that their approach is lightweight and orthogonal to existing weight-tuning methods, as it keeps the generative backbone frozen and exclusively optimizes the low-dimensional sampling policy. The dynamic teacher mechanism provides a theoretically grounded learning signal that forces the student to minimize its own discretization error, leading to sampling trajectories that are more consistent with the true continuous-time process and perceptually superior, particularly under strict low-NFE constraints.