Technical Summary: Split the Differences, Pool the Rest

Problem Statement

This work addresses Multi-Objective Imitation Learning (MOIL) within the framework of Multi-Objective Markov Decision Processes (MOMDPs). The core challenge is to recover policies lying on the Pareto front given demonstrations from multiple experts, where each expert optimizes a distinct utility function over a shared vector-valued reward.

The authors identify two critical failure modes in existing approaches:

1. Failure I (Sample Inefficiency): Treating each expert dataset independently (e.g., training separate Behavioral Cloning models for each) ignores the structural overlap between experts. Since Pareto-optimal policies often share substantial state-action agreements, independent learning requires each dataset size to scale linearly with the state space cardinality ($N_\ell = \Omega(|X|)$), leading to severe sample inefficiency.
2. Failure II (Pareto Violation): Naively pooling all expert trajectories into a single dataset and training a joint model (Single-Output Behavioral Cloning) can result in dominated policies. Theoretical analysis (Theorem 2.2) proves that the average policy of two Pareto-optimal experts is not necessarily Pareto-optimal. In non-linear environments, this aggregation causes the learned policy to "sag" into the interior of the return polytope, failing to optimize any coherent utility function even with infinite data.

Methodology: Multi-Output Augmented Behavioral Cloning (MA-BC)

To resolve these failures, the authors propose Multi-Output Augmented Behavioral Cloning (MA-BC). The algorithm is grounded in a new structural property of MOMDPs: the Pareto Path Existence (Theorem 2.1). This theorem establishes that any two neighboring Pareto-front policies differing in $k$ states can be connected by a sequence of $k$ intermediate Pareto-optimal policies, where each step differs by exactly one state-action pair.

Algorithmic Steps:

1. Partitioning: The algorithm identifies divergent states ($D_{div}$) where experts exhibit conflicting actions. Conversely, it defines a set of common states ($D_{common}$) where all experts agree on the action.
2. Data Curation: For each target expert $\ell$, the algorithm constructs a curated dataset $D_\ell \cup D_{common}$. This allows the learner to:
   • Use the specific expert's data ($D_\ell$) for divergent states.
   • Pool data from all experts for common states, effectively reusing samples where no conflict exists.
3. Training: $L$ separate policies are trained via Behavioral Cloning on these curated datasets.

Extension to Continuous Spaces:
For continuous state-action spaces, MA-BC is extended by augmenting each expert's dataset with compatible state-action pairs from other experts. Compatibility is defined by a spatial tolerance $\delta$ and the Lipschitz constant of the expert's policy, creating a bias-variance tradeoff controlled by $\delta$.

Key Contributions

1. Structural Characterization (Pareto Path): The paper introduces a novel geometric characterization of MOMDPs (Theorem 2.1), proving that Pareto-optimal policies are structurally similar and connected via single-state changes, even when multiple policies map to the same return vector (aliasing).
2. Algorithmic Solution (MA-BC): The authors introduce MA-BC, which systematically partitions divergent data while pooling non-conflicting data.
3. Theoretical Guarantees:
   • Convergence: Theorem 3.1 proves that MA-BC converges to $\epsilon$-Pareto-optimal policies. Crucially, the sample complexity for the joint dataset scales with the total number of states $|X|$, while the individual expert dataset sizes only need to scale with the number of divergent states $K$ (where $K \ll |X|$). This improves upon the $\Omega(|X|)$ requirement of independent learners.
   • Minimax Optimality: A novel lower bound (Theorem 3.2) is established for offline MOIL, demonstrating that MA-BC is minimax optimal. The bound highlights that the performance gain depends on the concentrability coefficient ($C_{\pi_{data}}$), which measures the overlap in state occupancy between experts.
4. Handling Unsplit Data: The framework is extended to scenarios where expert identities are latent (unsplit data), utilizing a Consistency Graph to cluster divergent state-action pairs (Appendix D).

Experimental Results

The authors validate MA-BC across four domains:

• Discrete Benchmarks: Deep Sea Treasure, Resource Gathering, and Slippery Y-Maze.
  • Results: Naive BC consistently fails to converge to the Pareto front (validating Failure II). Isolated BC converges but exhibits poor sample efficiency (validating Failure I). MA-BC converges to the Pareto front significantly faster, leveraging shared structural knowledge.
• Concentrability Analysis: Experiments show that as the overlap between expert trajectories increases (lowering the concentrability coefficient), the performance gap between MA-BC and isolated learners widens, aligning with theoretical predictions.
• Continuous Control: A 6-DOF quadcopter LQR task.
  • Results: The study demonstrates a "bias-data tradeoff." In low-data regimes, a larger pooling threshold $\delta$ improves performance by aggressively sharing data. In high-data regimes, a smaller $\delta$ is optimal to minimize bias. The algorithm successfully recovers the Economic expert policy with superior sample efficiency compared to baselines.

Significance and Claims

The paper claims to be the first to provide a provably efficient framework for offline imitation learning in MOMDPs. Its primary significance lies in bridging the gap between the theoretical necessity of Pareto optimality and the practical constraints of data efficiency.

The authors argue that standard aggregation strategies are fundamentally flawed in multi-objective settings because they destroy the geometric structure of the Pareto front. By leveraging the Pareto Path property, MA-BC achieves a "best of both worlds" outcome: it maintains the theoretical guarantees of single-expert imitation (avoiding dominated policies) while achieving the sample efficiency of pooled learning (utilizing shared state-action overlaps). The work establishes that data from multiple experts can indeed be used to efficiently learn Pareto-optimal policies, provided the data is partitioned based on behavioral conflicts rather than aggregated blindly or treated entirely independently.