Technical Summary: Drifting Preference Optimization for One-Step Generative Models

Problem Statement

One-step text-to-image generators (e.g., SD-Turbo, SDXL-Turbo) are highly attractive for deployment due to their ability to generate images in a single forward pass, enabling low-latency and resource-constrained applications. However, aligning these models with human preferences via post-training remains challenging.

Standard alignment methods, such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), typically rely on stochastic policy probabilities, density ratios, or denoising trajectories—structures that are unavailable or ill-defined in deterministic one-step generators. While directly backpropagating a differentiable reward through the generated image is an alternative, it requires the reward model to be differentiable. This approach becomes computationally prohibitive for large Vision-Language Model (VLM) based rewards and is impossible for black-box or discrete evaluators (e.g., rule-based correctness checks). Consequently, there is a need for a preference-finetuning method that:

1. Works with deterministic one-step generators.
2. Avoids backpropagation through the reward model.
3. Supports large, black-box, or non-differentiable rewards.
4. Preserves the single-forward-pass inference capability.

Methodology: Drifting Preference Optimization (DrPO)

The authors propose DrPO, an online preference-finetuning method that converts reward-ranked samples into a feature-space update direction without requiring reward gradients. The method draws inspiration from "Drifting Models," which replace explicit density estimation with feature-space drift fields.

Core Mechanism

DrPO operates by constructing a non-parametric dipole preference field and a reference drift field in the latent feature space.

1. Candidate Sampling and Ranking:
   For a given prompt, the current generator $g_\theta$ samples a batch of candidates. A target reward model $R$ (which can be non-differentiable) ranks these candidates. Based on this ranking, the method forms pairs of positive (high-scoring) and negative (low-scoring) samples.

2. Dipole Preference Field ($V_{pref}$):
   Instead of using the gradient of the reward function, DrPO synthesizes an update direction using a kernel-weighted attraction-repulsion mechanism.

   • Attraction: Positive samples pull the current feature representation toward them.
   • Repulsion: Negative samples push the representation away.
   • This is formalized as a dipole field where the update velocity is the difference between the mean-shift of positive samples and the mean-shift of negative samples in feature space. This field is local; points far from the observed samples receive negligible updates, reducing extrapolation errors.

3. Reference Drift ($V_{ref}$):
   To prevent the generator from drifting too far from its original distribution (mode collapse or loss of fidelity), a reference drift is estimated from a frozen base generator ($g_{ref}$). This acts as a stabilizing force, similar to KL regularization in standard RLHF, but estimated via sample-based drift rather than likelihood ratios.

4. Optimization Objective:
   The total update field is the sum of the preference field and the weighted reference field: $V_{DrPO} = V_{pref} + \lambda V_{ref}$.
   The generator is trained to minimize the distance between its current feature output and a detached target:
   $$z^* = \text{sg}(z + \eta V_{DrPO}(z))$$
   where $z$ is the feature representation of the generated image, $\text{sg}$ denotes the stop-gradient operator, and $\eta$ is a velocity scale. The loss is a simple mean squared error between the current features and the detached target.

Key Properties

• Reward Agnosticism: Since the reward model is only used to rank samples and form the positive/negative sets ($A^+$ and $A^-$), the reward function itself never needs to be differentiated. This allows the use of VLM-based rewards (e.g., HPSv3) and discrete task evaluators (e.g., GenEval).
• One-Step Inference: The training modifies the generator weights, but inference remains a single forward pass of the updated generator. No test-time optimization or search is required.
• Offline Extension: The authors propose a naive offline variant where preference pairs are drawn from a fixed dataset rather than online sampling, though they note this requires careful handling of off-policy sparsity.

Key Contributions

1. DrPO Algorithm: A novel online preference-finetuning method for deterministic one-step generators that converts reward-ranked samples into a detached drifting update, eliminating the need for reward backpropagation.
2. Dipole Preference Field: A non-parametric mechanism that constructs latent-space update directions from positive and negative samples, replacing traditional reward gradients with sample-based drifting updates.
3. Empirical Validation: Comprehensive evaluation on SD-Turbo and SDXL-Turbo across multiple reward models (PickScore, HPSv3, ImageReward, GenEval). DrPO demonstrates superior performance among reward-gradient-free baselines.
4. Efficiency Gains: Demonstration that DrPO significantly reduces training computation when using large reward models by removing the need for reward-model backpropagation.

Experimental Results

The authors evaluated DrPO on SD-Turbo and SDXL-Turbo using benchmarks including Pick-a-Pic v2, Parti-Prompts, and GenEval.

• Performance: On SDXL-Turbo, DrPO achieved the highest gains among reward-gradient-free one-step baselines (DPO1-step, PSO, GRPO1-step) in terms of PickScore, Aesthetic Score (AES), and ImageReward. It approached or matched the performance of reward-gradient-based methods (like DRaFT) while avoiding the computational cost of backpropagating through the reward.
• Human/VLM Preference: Pairwise evaluations using Qwen3-VL showed that DrPO-generated images were consistently preferred over baseline one-step methods and the base model, indicating better prompt fidelity and visual coherence.
• Non-Differentiable Rewards: DrPO successfully utilized GenEval scores (a discrete, non-differentiable metric for compositional constraints) to improve performance on tasks like counting, color placement, and object binding, where gradient-based methods fail.
• Efficiency: When trained with the large HPSv3 reward model, DrPO reduced the update time by 3.51× compared to a baseline that backpropagates through the reward, under matched effective batch settings.
• Ablations: Experiments confirmed that increasing the candidate pool size improves stability, and that latent-MAE features outperform raw latent features for the drift calculation. The method was found robust to kernel choices and velocity scales.

Significance and Claims

The paper claims that DrPO provides a practical solution for aligning one-step generative models with complex, expensive, or non-differentiable reward signals. By decoupling the reward evaluation from the optimization gradient, DrPO enables the use of state-of-the-art VLM-based reward models (like HPSv3) for fine-tuning fast generators without incurring the prohibitive computational cost of differentiating through them.

The authors position DrPO as a significant step toward making high-quality, low-latency image generation accessible for interactive applications where both speed and alignment with human preferences are critical. They also note that while their offline experiments are preliminary, the sample-based gradient synthesis approach offers a promising direction for extending preference optimization to offline settings where online sampling is not feasible.

The work does not claim to solve all alignment challenges; it acknowledges that the quality of the preference field is dependent on the feature extractor's ability to encode the specific attributes being optimized and that the method inherits the limitations of the underlying reward model used for ranking.