Technical Summary: Multi-Objective Preference Optimization (MOPO)

1. Problem Statement

Current approaches to aligning Large Language Models (LLMs) with human preferences, such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), predominantly rely on single-objective optimization. These methods typically collapse multiple human preferences (e.g., helpfulness, harmlessness, factuality, diversity) into a single scalar reward signal via weighted sums or heuristic scalarization.

The paper argues that this approach is fundamentally limited because:

1. Conflicting Objectives: Human preferences are often multi-dimensional and conflicting (e.g., maximizing helpfulness may degrade harmlessness).
2. Loss of Pareto Optimality: Scalarization hides trade-offs and fails to recover the Pareto front—the set of non-dominated solutions where no objective can be improved without worsening another.
3. Inference Rigidity: Existing multi-objective methods often require explicit user preference vectors at inference time (e.g., "0.6 helpful, 0.4 safe"), which users rarely articulate explicitly, or they rely on retraining for every weight configuration.

The core problem addressed is Multi-Objective Preference Alignment: learning a single policy that simultaneously balances multiple, potentially conflicting objectives based on pairwise preference data, without requiring explicit scalarization or user-provided weights at inference.

2. Methodology: MOPO

The authors propose Multi-Objective Preference Optimization (MOPO), an offline, constrained optimization framework that learns directly from pairwise preference data.

2.1 Formulation

MOPO frames the alignment problem as a Concave Constrained Optimization Problem (COP). Instead of maximizing a weighted sum of rewards, it maximizes a primary objective while enforcing lower-bound constraints on secondary objectives.

Given a reference policy $\pi_{ref}$, a primary preference function $p(\cdot)$, and a vector of secondary preference functions $q(\cdot)$, the problem is:
$$\begin{aligned} \max_{\pi} \quad & \mathbb{E}_{x, y, y'} [p(y \succ y' | x)] - \tau \text{KL}(\pi \parallel \pi_{ref}) \\ \text{s.t.} \quad & \mathbb{E}_{x, y, y'} [q(y \succ y' | x)] \geq \mathbf{b} \end{aligned}$$
where $\mathbf{b}$ represents tunable safety thresholds for the secondary objectives, and $\tau$ is a regularization parameter.

2.2 Algorithmic Components

To solve this without point-wise rewards, MOPO utilizes the following mechanisms:

• Importance Sampling Ratio ($\rho$): The policy is represented via the ratio $\rho(y) = \pi(y) / \pi_{ref}(y)$. The optimization is performed over $\rho$ to derive a closed-form solution for the optimal policy.
• Lagrangian Dual Formulation: The constrained problem is converted into a dual problem involving Lagrange multipliers $\lambda$. The optimal $\rho$ is derived as:
  $$\rho^*_\lambda(y) = \exp\left( \tau^{-1} \mathbb{E}_{y' \sim \mu} [p(y \succ y') + \lambda^T q(y \succ y')] - 1 \right)$$
  Here, $\lambda$ acts as a penalty that increases if the policy underperforms on secondary objectives.
• Robust Lower Bound Estimation: To handle estimation errors in finite datasets, MOPO does not constrain the raw empirical expectation directly. Instead, it constrains a lower bound of the preference estimates. This is achieved by solving an auxiliary adversarial optimization problem to find a distribution $\pi_k$ that minimizes the preference for objective $k$ within a KL-divergence budget ($\epsilon$). This yields a robust lower bound estimate used in the main optimization loop.
• Iterative Updates: The algorithm alternates between:
  1. Updating the lower-bound parameters ($\chi$) to tighten the constraint estimates.
  2. Updating the Lagrange multipliers ($\lambda$) to balance objectives.
  3. Updating the policy parameters ($\psi$) via importance-weighted behavioral cloning.

2.3 Practical Implementation

• Lagged Reference Policy: To prevent collapse and ensure stable progression toward the Pareto front, the reference policy $\pi_{ref}$ is updated periodically (every $t_0$ steps) to the current policy $\pi^{(t-t_0)}$, rather than remaining fixed.
• Adaptive Constraint Schedule: Since exact constraint thresholds $\mathbf{b}$ are unknown, they are adaptively set based on the performance of the previously optimized policy, scaled by a relaxation parameter $\beta$.
• Multi-Head Architecture: For inference, MOPO employs a shared transformer backbone with multiple policy heads. At inference time, the user selects a primary objective, and the corresponding head is used, allowing a single model to adapt to different preference priorities without retraining.

3. Key Contributions

1. MOPO Framework: Introduction of an offline, constrained optimization algorithm that optimizes for multiple objectives simultaneously, achieving Pareto optimality without collapsing preferences into a single scalar.
2. Theoretical Guarantees: Demonstration that the proposed constrained formulation recovers policies lying on the true Pareto front, unlike standard scalarization methods (DPO, RLHF) which fail to cover the full front.
3. Robustness to Estimation Error: The development of a lower-bound constraint mechanism that makes the policy robust to noise in preference estimation, avoiding constraint violations in deployment.
4. Empirical Validation:
   • Synthetic Benchmarks: MOPO exactly recovers the optimal policy and the Pareto front in synthetic settings where the ground truth is known.
   • Real-World LLMs: Experiments on 7B-parameter models (e.g., Zephyr-7b-beta, Llama-3.1-8B) on tasks like "Helpful Assistant" and "Reddit Summarization" show that MOPO achieves higher rewards and Pareto-dominates baselines (including RiC, PARM, MODPO, and DPO variants).
   • Stability: Ablation studies confirm the algorithm's stability across varying hyperparameters (constraint relaxation $\beta$, lag $t_0$, and regularization $\tau$).

4. Results

• Pareto Recovery: In synthetic experiments, MOPO successfully approximates the Pareto front, whereas DPO trained on single objectives or scalarized rewards misses significant portions of the front.
• Performance on LLMs: On the Helpful Assistant and Reddit Summary tasks, MOPO (specifically the Lagrangian variant, MOPO-Lag) consistently outperforms or matches state-of-the-art baselines across multiple objectives (helpfulness, harmlessness, humor, faithfulness).
• Scalability: The method scales from small models (phi-1.5) to larger 7B/8B models.
• Constraint Satisfaction: In verbosity-constrained tasks (controlling response length), MOPO successfully maintains response lengths within specified bounds while improving helpfulness.

5. Significance and Claims

The paper claims that MOPO addresses a critical gap in current alignment paradigms: the inability to handle multi-dimensional human preferences without arbitrary scalarization or user intervention at inference.

• Paradigm Shift: It moves beyond the "single reward signal" assumption of standard RLHF/DPO, offering a principled way to navigate trade-offs between conflicting goals (e.g., safety vs. helpfulness).
• Practical Utility: By recovering Pareto-optimal policies offline, MOPO enables the deployment of models that are robust and adaptable to diverse user needs without requiring explicit preference vectors or retraining for every new weight configuration.
• Future Directions: The authors modestly note that while empirical results are strong, a rigorous theoretical analysis of MOPO's convergence and optimality guarantees remains an important future direction.

The work suggests that for complex generative models, treating alignment as a constrained multi-objective problem is superior to treating it as a single-objective optimization with weighted rewards.