Relative Density Ratio Optimization for Stable and Statistically Consistent Model Alignment

1. Problem Statement

Aligning Large Language Models (LLMs) with human preferences is critical for safety and reliability. Existing methods generally fall into two categories, both of which have limitations:

• Model-Based Approaches (e.g., RLHF, DPO, KTO): These assume specific human preference models, such as the Bradley-Terry model or Prospect Theory. While powerful, these assumptions often fail to capture the true complexity of human preferences (e.g., cyclic preferences). Consequently, these methods lack statistical consistency, meaning there is no guarantee that the model will converge to the true human preference distribution as the sample size increases.
• Direct Density Ratio Optimization (DDRO): This approach avoids assuming a specific preference model by directly estimating the density ratio between preferred ($p_+$) and non-preferred ($p_-$) data distributions. While DDRO is statistically consistent, it suffers from training instability. Theoretically, the density ratio $p_-(y|x) / p_+(y|x)$ can diverge to infinity if the supports of the two distributions differ (i.e., if a response is possible under one distribution but impossible under the other), leading to gradient spikes and training failure.

The Core Challenge: How to achieve an alignment method that is both statistically consistent (converges to the true preference without model assumptions) and stable (avoids divergence during training).

2. Methodology: Relative Density Ratio Optimization (RDRO)

The authors propose RDRO, a novel alignment method that replaces the standard density ratio with a relative density ratio.

Key Concept: Relative Density Ratio

Instead of modeling the ratio $g^*(y|x) = p_-(y|x) / p_+(y|x)$, RDRO models the ratio between the preferred distribution and a mixture of the preferred and non-preferred distributions:
$$r^*(y|x) \equiv \frac{p_+(y|x)}{p_{ref}(y|x)} = \frac{p_+(y|x)}{\alpha p_+(y|x) + (1-\alpha)p_-(y|x)}$$
Where:

• $p_{ref}$ is the reference policy (a mixture of $p_+$ and $p_-$).
• $\alpha \in (0, 1)$ is a hyperparameter representing the probability of a response being preferred.

Theoretical Advantages

1. Boundedness: Unlike the standard density ratio which can range from $[0, \infty)$, the relative density ratio is strictly bounded: $r^*(y|x) \in [0, 1/\alpha]$. This prevents the ratio from diverging even when the supports of $p_+$ and $p_-$ differ, ensuring training stability.
2. Statistical Consistency: The method retains the statistical consistency of DDRO. As the number of samples increases, the policy $p_\theta$ converges to the true preference $p_+$.
3. Tighter Convergence Guarantees: The authors prove that the estimation error bound for RDRO is significantly tighter than that of DDRO. Specifically, the bound depends on $1/\alpha$ rather than $1/m_+$ (where $m_+$ is the minimum probability in the support of $p_+$, often very small), and the Lipschitz constant of the loss function is smaller due to the bounded nature of $r^*$.

Optimization Objective

The method minimizes the Bregman divergence between the true relative density ratio $r^*$ and the modeled ratio $r_\theta$. Using a logistic regression formulation (where $f(t) = t \log t - (1+t)\log(1+t)$), the loss function is derived as:
$$L_{RDRO}(\theta) = \mathbb{E}_{p_+}[(1+\alpha)\phi(T_\theta) - T_\theta] + \mathbb{E}_{p_-}[(1-\alpha)\phi(T_\theta)]$$
Where:

• $T_\theta(x, y) = \log p_\theta(y|x) - \log p_{ref}(y|x)$ is the log-odds ratio.
• $\phi(t) = \log(1 + e^t)$ is the softplus function.
• No KL Regularizer Needed: Unlike DDRO, which requires an explicit KL divergence regularizer to prevent catastrophic forgetting, the parameter $\alpha$ in RDRO naturally acts as a regularization parameter, controlling how much $p_\theta$ can deviate from $p_{ref}$.

3. Key Contributions

1. Novel Algorithm: Proposes RDRO, the first alignment method that is both statistically consistent and stable without assuming a specific human preference model.
2. Theoretical Proof:
   • Proves that the relative density ratio is bounded, preventing divergence.
   • Derives a convergence rate bound showing that RDRO yields significantly tighter error bounds than DDRO, particularly in scenarios where the density ratio would otherwise explode.
3. Empirical Validation: Demonstrates effectiveness across multiple model sizes (Qwen 2.5 1.5B/3B, Llama 3 3B/8B) and datasets.

4. Experimental Results

The authors evaluated RDRO against KTO (a state-of-the-art preference model-based method) and DDRO (the previous density-ratio-based method) using the AlpacaEval LC (Length-Controlled) Win Rate metric.

• Datasets:
  • UF-G: A challenging dataset where prompts often have only one labeled response (preferred OR non-preferred), making alignment difficult.
  • MIX-14K: A dataset derived from DPO-Mix where every prompt has both preferred and non-preferred responses.
• Performance on UF-G (Harder Task):
  • RDRO consistently outperformed both KTO and DDRO across all model sizes.
  • Notable improvements: On Llama-8B, RDRO achieved 17.31% win rate vs. 15.66% for DDRO and 9.34% for KTO.
  • Statistical significance was confirmed via t-tests for larger models.
• Performance on MIX-14K:
  • RDRO performed comparably to DDRO and KTO, showing it does not sacrifice performance on easier, triplet-based datasets.
• Hyperparameter Sensitivity:
  • The method is robust to the choice of $\alpha$. Performance remained stable across a wide range of $\alpha$ values, though setting $\alpha$ to the fraction of preferred samples in the dataset yielded optimal results.

5. Significance

This paper addresses a fundamental gap in LLM alignment theory and practice:

• Solving the Stability-Consistency Trade-off: Prior methods forced a choice between statistical consistency (DDRO) and training stability (RLHF/DPO). RDRO achieves both.
• Robustness to Data Sparsity: By bounding the density ratio, RDRO is less susceptible to the "support mismatch" problem, which is common in real-world datasets where preferred and non-preferred responses may occupy different regions of the output space.
• Simplified Training: The elimination of the explicit KL regularizer simplifies the training objective while maintaining regularization through the mixture parameter $\alpha$.

In conclusion, RDRO provides a theoretically grounded, stable, and high-performing alternative for aligning LLMs with human preferences, particularly in scenarios where data is noisy or incomplete.