Technical Summary: Arena as Offline Reward (ArenaPO)

Problem Statement

While Reinforcement Learning from Human Feedback (RLHF) has effectively aligned Text-to-Image (T2I) diffusion models with human preferences, it suffers from computational inefficiency due to the requirement of training and deploying an explicit reward model with online inference. Direct Preference Optimization (DPO) has emerged as a more efficient alternative by leveraging static offline datasets without explicit reward modeling. However, standard DPO is limited to binary feedback (chosen vs. rejected pairs), which results in coarse-grained optimization. In practice, the quality differences between image pairs vary significantly; treating all "chosen" images as equally superior to "rejected" ones ignores the magnitude of preference, potentially leading to suboptimal alignment. The core challenge is how to incorporate fine-grained preference signals into the efficient DPO framework without relying on an explicit, computationally expensive reward model.

Methodology: ArenaPO

The authors propose ArenaPO, a framework that extracts fine-grained offline rewards from a "Model Arena" to enhance DPO training. The method operates in three distinct stages:

1. Model Arena Construction

Instead of treating images in isolation, the authors construct a "Model Arena" where each image's source model is a competitor.

• Probabilistic Modeling: Following Thurstone's Case V theory, each model's capability is modeled not as a fixed point but as a Gaussian distribution $\theta \sim \mathcal{N}(\mu, \sigma^2)$, where $\mu$ represents the mean capability and $\sigma$ represents uncertainty.
• Bayesian Updating: The dataset is viewed as a series of pairwise comparisons. By traversing the annotated pairwise preferences, the capability distribution of each model is iteratively updated using Bayesian inference. The binary label (win/loss) serves as the likelihood function to update the prior distribution, converging to a posterior distribution that characterizes the model's true capability.

2. Latent-Variable Inference for Quality Gap Estimation

For any given image pair $(y_w, y_l)$ in the dataset, the method estimates the absolute quality gap without a reward model.

• Formulation: The quality of an image is treated as a single sample drawn from its source model's capability distribution. The problem is formulated as estimating the posterior expectation of the quality difference $Q_D = Q_w - Q_l$ given the observed event $Q_w > Q_l$.
• Truncated Normal Distribution: Since the difference between two independent Gaussian variables is also Gaussian, and the observation imposes a one-sided constraint ($Q_D > 0$), the authors utilize the properties of the truncated normal distribution.
• Closed-Form Solution: The estimated quality gap $Q_D^*$ is computed as the conditional expectation $E[Q_D | Q_D > 0]$. This value incorporates both the prior mean difference ($\mu_w - \mu_l$) and a correction term derived from the uncertainty ($\sigma_w, \sigma_l$) and the observed binary preference. This $Q_D^*$ serves as a fine-grained, instance-specific reward signal.

3. Fine-Grained DPO Training

The estimated quality gap is integrated into the Diffusion-DPO objective.

• Margin-Based Bradley-Terry Model: Instead of enforcing a simple binary ordering, the preference probability is modeled with a margin: $p(y_w \succ y_l) = \sigma(r(y_w) - r(y_l) - Q_D^*)$.
• Loss Function: The standard DPO loss is modified to include $Q_D^*$ as a scaling factor (weighted by a coefficient $\gamma$). This encourages the policy to not only learn which image is better but also how much better it is, matching the magnitude of the preference gap.
• Efficiency: The entire pipeline is offline. The Arena construction and quality gap estimation are computed prior to training, requiring no additional online inference or reward model training, thus maintaining the computational efficiency of DPO.

Key Contributions

1. ArenaPO Framework: The proposal of a method that leverages Arena scores as offline rewards to provide refined feedback for DPO, achieving both the rich reward signals of RLHF and the efficiency of DPO without an explicit reward model.
2. Latent-Variable Inference: The introduction of a novel inference method based on the truncated normal distribution to estimate absolute quality gaps between image pairs using only model capability distributions and binary preferences.
3. Empirical Superiority: Extensive experiments on Pick-a-Pic v2 and HPD v3 datasets demonstrating that ArenaPO consistently outperforms existing baselines (including Diffusion-DPO, MaPO, DSPO, and SDPO) across multiple evaluation metrics (PickScore, HPS, Aesthetic, CLIP, ImageReward).

Experimental Results

The authors evaluated ArenaPO on Stable Diffusion 1.5 and Stable Diffusion XL using Pick-a-Pic v2 and HPD v3 datasets.

• Performance Gains: ArenaPO achieved state-of-the-art results across all tested benchmarks. For example, on the Pick-a-Pic v2 dataset with Stable Diffusion 1.5, ArenaPO improved the PickScore from 0.2143 (SDPO) to 0.2151 and ImageReward from 0.5546 to 0.6063.
• Win-Rate: In head-to-head comparisons against the original Stable Diffusion 1.5, ArenaPO achieved a win-rate of 79.0%, significantly outperforming other DPO variants.
• Ablation Studies:
  • Fine-Grained vs. Uniform Margin: Replacing the adaptive $Q_D^*$ with a fixed uniform margin resulted in significantly lower performance, validating the necessity of fine-grained, adaptive rewards.
  • Scalability: Combining ArenaPO with other advanced methods (e.g., DSPO) yielded further performance gains, indicating the approach is decoupled and compatible with other optimization strategies.
  • Hyperparameter Sensitivity: The scaling coefficient $\gamma$ was found to be relatively insensitive, with optimal performance at $1 \times 10^{-3}$.

Significance and Claims

The paper claims that ArenaPO effectively bridges the gap between the efficiency of DPO and the richness of RLHF. By modeling model capabilities as distributions and using latent-variable inference, the method extracts fine-grained preference information from existing binary datasets without the overhead of training a reward model. The authors assert that this approach leads to more effective alignment, allowing models to learn not just the direction of preference but the intensity of that preference. The method is presented as a practical, scalable solution for improving T2I diffusion models in real-world applications where computational resources for online reward modeling are limited.