🤖 AI

Multi-Objective Constraint Inference using Inverse reinforcement learning

This paper introduces Multi-Objective Constraint Inference (MOCI), a novel framework that effectively extracts shared constraints and individual preferences from heterogeneous expert demonstrations with conflicting objectives, outperforming existing baselines in predictive accuracy while maintaining computational efficiency.

Original authors: Syed Ihtesham Hussain Shah, Floris den Hengst, Aneta Lisowska, Annette ten Teije

Published 2026-05-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Syed Ihtesham Hussain Shah, Floris den Hengst, Aneta Lisowska, Annette ten Teije

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to figure out the rules of a game and the secret motivations of the players, but you can only watch them play. You can't ask them questions, and you don't have a rulebook. You just have a pile of video recordings of different people playing the same game.

This paper introduces a new detective tool called MOCI (Multi-Objective Constraint Inference). Here is how it works, broken down into simple concepts:

The Problem: One Size Doesn't Fit All

Most previous detective tools had two big problems:

They assumed everyone was the same: They thought all the players in the videos were following the exact same strategy and had the exact same goals.
They got confused by "unvisited" areas: If a player never stepped on a specific square on the board, the old tools couldn't tell if that square was a "forbidden trap" or just a place the player simply didn't like.

In the real world, this is like watching a group of drivers. Some are aggressive, some are cautious. They all have to obey the same traffic laws (hard constraints like red lights and walls), but they have different personal preferences (some want the fastest route, others want the most scenic route). Old tools tried to force all these different drivers into a single "average" driver, which didn't work well.

The Solution: MOCI (The Smart Detective)

The authors created MOCI to solve this by doing two things at once:

Sorting the Players: It looks at the messy pile of videos and automatically groups them. It realizes, "Ah, these three videos show a 'Grass-Lover' who avoids rocks, and these two show a 'Rock-Lover' who avoids grass." It separates the players based on their unique tastes.
Finding the Invisible Walls: Once it knows who likes what, it looks at the whole group to find the rules everyone follows. If no one (not the Grass-Lover, not the Rock-Lover) ever steps on the blue water tiles, MOCI concludes, "Those blue tiles must be forbidden walls."

The "Gridworld" Experiment

To test this, the researchers created a digital game board (a Gridworld) with different types of terrain:

Grass: Some players love it.
Rocks: Other players love it.
Water: Nobody goes here. It's a hard constraint (a wall).

They mixed up videos of the "Grass-Lovers" and "Rock-Lovers" so the computer didn't know who was who. MOCI successfully:

Figured out that there were two different types of players.
Learned that the Grass-Lover gets a "bonus" for walking on grass.
Learned that the Rock-Lover gets a "bonus" for walking on rocks.
Correctly identified the Water tiles as forbidden zones, even though the players never explicitly said, "I can't go there."

The "Hallucination" Warning

The paper admits one small flaw. If the players never visit a specific corner of the map, MOCI might get a little paranoid and think, "Nobody went there, so it must be a wall!" It calls this a "false positive." However, the paper shows that if you give the detective more videos (more data), it stops guessing and becomes much more accurate.

Why It's Better Than the Competition

The authors compared MOCI to two other famous detective tools (MLCI and ICRL):

Accuracy: MOCI was much more accurate at guessing the rules and preferences (scoring a 0.027 error rate vs. 0.25 and 0.36 for the others).
Speed: While MOCI does more work than the simplest tool, it is still very fast. It's not a slow, clunky machine; it's efficient enough to be practical.
Flexibility: Unlike the others, MOCI can handle a crowd of different people with different goals. The others can only handle a crowd of identical robots.

The Bottom Line

MOCI is a new way to teach computers to understand safety rules and personal preferences by watching a mix of different people. It separates the "universal rules" (like don't walk into water) from "personal tastes" (like I prefer grass), doing it faster and more accurately than previous methods.

Note: The paper mentions that this technology could eventually be used in healthcare to help doctors balance shared safety rules with individual patient preferences, but the actual experiments in this paper were strictly limited to the digital grid game.

Technical Summary: Multi-Objective Constraint Inference using Inverse Reinforcement Learning

Problem Statement

Inverse Reinforcement Learning (IRL) aims to infer an agent's underlying objectives by observing expert demonstrations. While traditional approaches often model intent solely through a learned reward function, this can lead to overly complex and brittle models. A more robust alternative involves jointly learning a simple reward function and a set of hard constraints (e.g., physical barriers or safety boundaries).

However, existing constraint inference methods suffer from two restrictive assumptions that limit their applicability to real-world scenarios:

Homogeneity: They assume demonstrations are generated by a single expert type or multiple experts with identical objectives.
A Priori Knowledge: They often assume the expert's reward function is completely known beforehand.

In reality, observed data typically stems from a heterogeneous population where experts share common environmental constraints (e.g., traffic laws, physical walls) but possess distinct, private preferences (e.g., aggressive vs. cautious driving, or preferences for specific terrains). Current methods struggle to disentangle these shared constraints from individual preferences, often failing to capture diverse behaviors or suffering from computational inefficiencies.

Methodology: Multi-Objective Constraint Inference (MOCI)

To address these challenges, the authors propose Multi-Objective Constraint Inference (MOCI), a framework designed to jointly recover shared hard constraints and individual preference weights from unlabeled, heterogeneous expert trajectories.

Core Framework

MOCI operates within a Constrained Multi-Objective Markov Decision Process (CMOMDP) setting. It assumes:

A shared state-action feature space $\phi(s, a)$ .
A shared set of hard constraints $C$ (forbidden states) applicable to all experts.
$K$ latent expert types (clusters), each characterized by a unique preference weight vector $w_k$ .

The algorithm employs an Expectation-Maximization (EM) approach to maximize the joint log-likelihood of the dataset $D$ :

E-Step (Clustering): The algorithm fixes the current estimates of constraints $C$ , weights $\{w_k\}$ , and cluster priors $\{\pi_k\}$ . It computes the posterior probability (responsibility) $\gamma_{i,k}$ that a specific trajectory $\xi_i$ was generated by expert type $k$ . This effectively clusters the unlabeled demonstrations based on their likelihood under different preference models.
M-Step (Parameter Update):
- Priors: Updated as the empirical mean of the responsibilities.
- Reward Weights: For each cluster, the preference weights $w_k$ are updated via gradient ascent. The gradient is weighted by the responsibility $\gamma_{i,k}$ , ensuring that the reward function adapts to the trajectories probabilistically assigned to that cluster using Maximum Entropy IRL principles.
- Constraints: The shared constraint set $C$ is updated via a greedy search. The algorithm evaluates candidate states (those not visited in the dataset) by calculating the joint log-likelihood if the candidate were added to $C$ . A candidate is added only if it increases the likelihood significantly. This process is regularized by a threshold $d_{DKL}$ on the Kullback-Leibler divergence to prevent overfitting to noise or unvisited states.

Computational Complexity

The theoretical computational cost of MOCI is dominated by the state-action space size, horizon, and the number of iterations. The complexity is derived as $O(|S| \cdot |A| \cdot H)$ , where $|S|$ is the number of states, $|A|$ is the number of actions, and $H$ is the trajectory horizon. In a grid-world environment, this scales quadratically with the grid dimensions.

Key Contributions

Novel Framework: Introduction of MOCI, the first framework to jointly infer shared hard constraints and individual multi-objective preferences from heterogeneous, unlabeled demonstrations.
Algorithmic Innovation: Development of an EM-based algorithm that iteratively clusters trajectories and updates both reward weights and constraint sets, effectively separating individual motivations from environmental limitations.
Theoretical Analysis: Provision of a detailed computational complexity analysis demonstrating the algorithm's scalability.
Empirical Validation: Demonstration of the method's ability to recover ground-truth constraints and distinct preference profiles in a controlled multi-objective GridWorld environment.

Experimental Results

The authors evaluated MOCI on a simulated heterogeneous GridWorld environment featuring two expert types: a "Grass-Lover" and a "Rock-Lover," both navigating around shared "Water" (hard constraint) tiles.

Constraint Recovery: MOCI successfully identified the ground-truth water tiles as hard constraints. It also noted a limitation where unvisited states were occasionally inferred as constraints (false positives) due to lack of data coverage, a behavior mitigated by tuning the divergence threshold.
Preference Learning: The algorithm accurately recovered the distinct reward weights for both experts, correctly identifying the Grass-Lover's preference for grass and aversion to rocks/water, and the Rock-Lover's opposite preferences.
Performance Comparison: Compared to baselines Maximum Likelihood Constraint Inference (MLCI) and Inverse Constrained Reinforcement Learning (ICRL):
- Accuracy: MOCI achieved a significantly lower Constraint Mean Squared Error (CMSE) of 0.027, compared to 0.25 for MLCI and 0.36 for ICRL.
- Efficiency: MOCI maintained a competitive runtime (0.69s) compared to MLCI (0.29s) and was substantially faster than ICRL (8.90s).
- Capabilities: Unlike baselines restricted to homogeneous data without preference learning, MOCI successfully handled heterogeneous data and explicitly learned preferences.
Scalability: Runtime analysis showed that execution time scales manageably with grid size and trajectory length, validating the theoretical complexity.

Significance and Claims

The paper positions MOCI as an accurate, flexible, and computationally practical approach for real-world constraint inference and preference learning. The authors claim that MOCI establishes a new capability to model diverse, potentially conflicting behaviors while maintaining safety boundaries. By successfully separating shared constraints from individual preferences, the framework offers a robust solution for aligning reinforcement learning agents with safety guidelines in complex environments where expert data is varied.

The study concludes that while MOCI currently requires the number of expert types ( $K$ ) to be predefined and focuses strictly on hard constraints, it provides a foundational step toward safer, more adaptable AI systems capable of learning from diverse human behaviors. Future work suggested by the authors includes replacing fixed clustering with Bayesian non-parametric methods to automate the discovery of $K$ and extending the framework to soft constraints and real-world healthcare applications.