Technical Summary: Towards General Preference Alignment: Diffusion Models at Nash Equilibrium

Problem Statement

Reinforcement Learning from Human Feedback (RLHF) and its computationally efficient variant, Direct Preference Optimization (DPO), have become standard for aligning text-to-image (T2I) diffusion models with human preferences. However, existing preference-based methods for diffusion alignment largely inherit the RLHF paradigm, relying on implicit reward signals induced by the Bradley–Terry (BT) model. The authors argue that the BT model's assumption of transitivity (if A > B and B > C, then A > C) fails to capture the full complexity of human preferences, which can be intransitive (e.g., rock-paper-scissors cycles). Furthermore, current methods often convert human preferences into a reward-induced function before optimization, rather than modeling the preference relation directly. This paper posits that a game-theoretic perspective, specifically Nash Learning from Human Feedback (NLHF), offers a more robust framework for diffusion alignment that avoids the restrictive transitivity assumptions of the BT model.

Methodology: Diffusion Nash Preference Optimization (Diff.-NPO)

The authors propose Diff.-NPO, a general preference optimization framework derived from a Nash equilibrium perspective. Instead of maximizing a BT-induced likelihood, Diff.-NPO formulates alignment as a two-player zero-sum game between the current policy ($\pi$) and a previous policy ($\pi_{prev}$), regularized by a reference policy ($\pi_{ref}$).

Core Framework

1. Game-Theoretic Formulation: The objective is defined as a minimax game where the "max-player" (current policy) attempts to generate samples preferred over those of the "min-player" (previous policy), while both are regularized to stay close to a reference policy ($\pi_{ref}$) to prevent distributional drift.
2. Online Mirror Descent (OMD): To ensure stable learning dynamics and avoid the instability of greedy best-response updates, the method employs Optimistic Mirror Descent with entropy regularization (Hedge). This leads to a self-play objective where the policy update is a best response to the previous policy, constrained by the reference policy.
3. Loss Function Derivation:
   • The method lifts image-level preferences to the reverse trajectory space.
   • It derives a pairwise logistic loss based on the log-ratios of trajectory probabilities.
   • The final objective, $L_{Diff.-NPO}$, contrasts the current policy ($p_\theta$) against both the reference policy ($p_{ref}$) and the previous policy ($p_{prev}$).
   • The loss is formulated as:
     $$L_{Diff.-NPO} = -\mathbb{E} \left[ \log \sigma \left( \beta \left[ \delta_\theta - \frac{\tau}{\eta}\delta_{ref} - \frac{\eta-\tau}{\eta}\delta_{prev} \right] \right) \right]$$
     Where $\delta$ terms represent the difference in implicit rewards (noise prediction errors) between preferred ($x^+$) and rejected ($x^-$) samples.
   • The hyperparameter $\gamma = \tau/\eta$ controls the interpolation between the reference policy (DPO-style, $\gamma=1$) and the previous policy (pure self-play, $\gamma=0$).

Key Distinctions from Existing Methods

• vs. Diffusion-DPO: While Diffusion-DPO is offline and contrasts the current policy solely against a fixed reference, Diff.-NPO is inherently online and self-play oriented, evaluating updates against samples generated by the current/previous policy.
• vs. INPO (Iterative Nash Policy Optimization): Unlike INPO, which uses a squared loss justified by specific population assumptions that may not hold in practical oracle-based ranking, Diff.-NPO utilizes a pairwise-logistic loss. This is argued to be more stable when preference pairs are deterministically ordered by an oracle (ranking generated samples) rather than sampled from a probabilistic preference distribution.

Key Contributions

1. Theoretical Foundation: The paper bridges the gap between Nash Learning from Human Feedback (NLHF) and diffusion model alignment, formulating diffusion alignment as a general preference game that avoids the Bradley–Terry assumption.
2. Algorithm Proposal: The introduction of Diff.-NPO, a novel on-policy general preference optimization algorithm. It unifies standard DPO and self-play paradigms within a single formulation, where the regularization coefficient interpolates between these regimes.
3. Explicit Modeling Assumptions: The framework makes the assumptions behind preference optimization explicit, interpreting the objective as fitting a target distribution constructed from the reference policy, the previous policy, and the preference signal.
4. Empirical Validation: Extensive experiments on Stable Diffusion 1.5 (SD1.5) and SDXL using the Pick-a-Pic, Parti-Prompts, and HPSV2 datasets demonstrate the method's effectiveness.

Experimental Results

The authors evaluated Diff.-NPO against baselines including Online-SFT, Online-DPO, SPIN, and Seppo.

• Win Rates: Diff.-NPO consistently achieved the highest average win rates across all evaluation datasets (Pick-a-Pic, Parti-Prompts, HPSV2) for both SD1.5 and SDXL.
  • On SDXL with Parti-Prompts, Diff.-NPO improved the ImageReward win rate by 8.6% and the Aesthetic Score win rate by 10.8% relative to Online-Diff-DPO.
  • It outperformed Online-DPO on Pick-a-Pic and HPSV2 as well.
• Automatic Metrics: The method showed superior performance in automatic metrics (PickScore, HPSV2, CLIP, ImageReward, Aesthetic Score) compared to baselines.
• Ablation Studies:
  • $\tau/\eta$ Ratio: The study confirmed that an intermediate value (specifically $\tau/\eta = 1/2$) yields the best performance, validating the hypothesis that relying solely on the reference policy (DPO) or solely on the previous policy (pure self-play) is suboptimal.
  • Loss Function: Replacing the pairwise-logistic objective with a squared distance objective (as used in INPO) resulted in a substantial performance drop, supporting the design choice of the logistic loss for deterministic oracle rankings.
• Qualitative Analysis: Visual comparisons indicated that Diff.-NPO produces images with better semantic alignment, anatomical correctness (e.g., full-body portraits), and visual coherence compared to baselines.

Significance and Claims

The paper claims that Diff.-NPO provides a simple and stable approach to online diffusion alignment by directly optimizing relative preference probabilities through a self-play-style objective with reference-policy regularization.

The authors assert that Nash-style online learning is a promising alternative to reward-centric formulations for diffusion-model alignment. By moving away from the transitive assumptions of the Bradley–Terry model, the method offers a more general perspective on preference alignment. The work establishes a foundation for future research on Nash-style learning for diffusion models, demonstrating that unifying DPO and self-play through a game-theoretic lens can yield superior alignment quality.

The paper modestly notes a limitation: the current preference construction relies on an average rank of five scoring models, and incorporating a broader set of scoring models could provide a more comprehensive signal. Future work is planned to extend the framework to stronger online data-collection strategies and broader diffusion generation and editing tasks.