Technical Summary: A Single Deep Preference-Conditioned Policy for Learning Pareto Coverage Sets

Problem Statement

Multi-Objective Reinforcement Learning (MORL) aims to optimize multiple, often competing, reward signals. The central object in MORL is not a single optimal policy but a set of Pareto-optimal trade-offs (the Pareto front). A prominent direction involves preference-conditioned learning, where a single deep model outputs diverse policies based on a user-specified preference vector $\omega$.

However, existing approaches face a "linear–nonlinear tension":

1. Linear Scalarization: Methods targeting the Convex Coverage Set (CCS) often rely on linear weighted sums. While mathematically clean, they can only recover vertex solutions of the Pareto front. For a fixed preference, the optimal objective vector may be non-unique (e.g., a whole supporting face), leading to ambiguous targets and discontinuous policy switching as preferences change. Furthermore, they struggle to provide dense coverage of non-vertex trade-offs.
2. Nonlinear Scalarization: Approaches targeting the Pareto Coverage Set (PCS) can reach non-vertex solutions but often require complex problem formulations, strong regularization, or heuristics that may deviate from the true Pareto front. Additionally, the uniqueness and continuity of the mapping from preference to solution remain unclear under general nonlinear scalarizations, making stable "preference sweeping" difficult.

This paper addresses the challenge of learning a single deep policy that provides dense, stable, and continuous coverage of the Pareto front under nonlinear scalarization, specifically within the context of tabular Multi-Objective Markov Decision Processes (MOMDPs) before extending to deep settings.

Methodology

1. Theoretical Foundation: Smooth Tchebycheff (STCH) Scalarization

The authors adopt the Smooth Tchebycheff (STCH) utility function as a monotone utility for scalarization. Unlike the non-smooth Tchebycheff norm, STCH uses a log-sum-exp approximation controlled by a smoothing parameter $\tau$.

• Uniqueness and Optimality: For a discounted MOMDP with a finite state/action space, the attainable objective set $\mathcal{J}$ is a convex polytope. The authors prove that for any strictly positive preference vector $\omega$, the STCH-induced optimal objective vector $J_\omega$ is unique and Pareto-optimal.
• Lipschitz Continuity: Crucially, they prove that the mapping $\omega \mapsto J_\omega$ is Lipschitz continuous on the interior of the preference simplex. This theoretical guarantee ensures that small changes in preference result in small, stable changes in the target objective, justifying the use of preference sweeping for dense Pareto-front coverage.

2. Algorithm: Concave Mirror Descent Policy Iteration (CMDPI)

To solve the resulting nonlinear scalarized problem without altering the objective function, the authors formulate the optimization over occupancy measures ($\mu$).

• Mirror Descent Formulation: They derive a mirror descent algorithm in the occupancy-measure space using the negative conditional entropy as the distance-generating function. This yields an $O(1/k)$ convergence rate for objective suboptimality.
• Policy Iteration Interpretation: By leveraging the correspondence between Bregman divergence in occupancy space and KL-divergence in policy space, the update is shown to be equivalent to solving a KL-regularized MDP where the previous policy acts as a reference.
  • Soft Bellman Evaluation: The policy evaluation step solves a soft Bellman fixed point.
  • Multiplicative Policy Improvement: The policy improvement step updates the policy via exponential reweighting (multiplicative weights) based on the soft Q-values.
• Finite-Step Continuity: A key theoretical contribution is the proof that, in the tabular setting, the policy $\pi_k$ at any finite iteration $k$ remains continuous with respect to the preference $\omega$, provided the initialization is continuous and has full support. This contrasts with previous single-network approaches that often assume such regularity without verification.

3. Deep Implementation: PCSAC

The authors instantiate the CMDPI update into a deep, model-free and model-based Actor-Critic algorithm named PCSAC (Pareto Coverage Soft Actor-Critic).

• Architecture: The critic $Q_\phi(s, a, \omega)$ outputs an $m$-dimensional vector, and the actor $\pi_\theta(s, \omega)$ outputs a distribution over actions.
• Training: The algorithm uses off-policy optimization similar to Soft Actor-Critic (SAC). The critic minimizes the squared Bellman residual for the vector target. The actor is updated by minimizing the KL divergence to a target distribution proportional to $\pi_k \exp(Q/\alpha)$, effectively implementing the mirror descent step.
• Utopia Point Estimation: Since the utopia point $I$ (required for STCH) is unknown in deep settings, the authors propose an online estimation procedure using exponential moving averages of reward percentiles.

Key Contributions

1. Theoretical Guarantees for STCH: Established that under STCH scalarization in tabular MOMDPs, the optimal objective vector is unique and Pareto-optimal, and the preference-to-solution map is Lipschitz continuous. This provides a principled basis for dense Pareto-front coverage via preference sweeping.
2. CMDPI Algorithm: Derived an $O(1/k)$ convergent mirror-descent algorithm in occupancy-measure space that solves the nonlinear scalarized problem without objective modification.
3. Policy-Iteration Equivalence & Continuity: Rewrote the update in an equivalent policy-iteration form, providing a theoretical motivation for deep preference-conditioned learning, including the proof of $\omega$-continuity at finite iterations.
4. Empirical Performance: Demonstrated that the proposed methods achieve superior Pareto-front approximation quality (measured by Hypervolume) and expected utility performance compared to recent baselines (LS, CAPQL, PreCo, C-MORL) on both discrete and continuous control tasks.

Experimental Results

The authors evaluated their methods on eight discrete-action and eight continuous-action MO-Gymnasium tasks.

• Tabular Planning: In a controlled toy MOMDP, CMDPI recovered Pareto-optimal solutions with significantly more uniform and dense coverage compared to linear scalarization (which only hit vertices) and CAPQL (which showed biased coverage).
• Discrete-Action Tasks: On eight MO-Gymnasium benchmarks, CMDPI achieved the best average rank for Hypervolume (HV) and Expected Utility Metric (EUM). PCSAC also outperformed linear scalarization and CAPQL in HV.
• Continuous-Action Tasks: On eight MuJoCo tasks, the model-based variant PCSAC-MB achieved the best average rank on both HV and EUM, outperforming linear-scalarization baselines (CAPQL-MB) and other state-of-the-art MORL methods. The model-free variant PCSAC-MF also showed strong performance, particularly in HV.
• Robustness: Sensitivity analyses indicated that the methods are robust to variations in the utopia point estimation and the temperature/smoothing parameters.

Significance and Claims

The paper claims to bridge the gap between the theoretical stability of linear scalarization and the coverage capabilities of nonlinear methods. By proving the uniqueness and Lipschitz continuity of the STCH-induced solution, the authors provide a rigorous foundation for using a single deep network to cover the Pareto front densely and stably.

The significance lies in:

• Principled Dense Coverage: Moving beyond heuristic preference sweeping to a method with theoretical guarantees of stability and coverage density.
• Algorithmic Clarity: Providing a clear connection between mirror descent, KL-regularized MDPs, and nonlinear MORL, offering a new perspective on policy iteration.
• Practical Efficacy: Demonstrating that these theoretical insights translate to state-of-the-art performance in both discrete and continuous control domains, particularly in sample efficiency for model-based settings.

The authors modestly note that their theoretical analysis is currently limited to the tabular setting and that extending these guarantees to function approximation remains an open direction for future work. They also acknowledge that the analysis relies on smooth, strictly concave utilities, leaving the behavior under other utility classes unexplored.