🤖 machine learning

Quantifying Potential Observation Missingness in Inverse Reinforcement Learning

This paper addresses the issue of missing observations in real-world behavioral datasets that can mislead Inverse Reinforcement Learning (IRL) by proposing an algorithm to quantify the minimal perturbations required to make expert actions appear optimal, thereby helping to identify potential data gaps in applications like healthcare.

Original authors: Leo Benac, Abhishek Sharma, Alihan Huyuk, Finale Doshi-Velez

Published 2026-05-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Leo Benac, Abhishek Sharma, Alihan Huyuk, Finale Doshi-Velez

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to figure out why a master chef always adds a specific pinch of salt at a certain moment. You have a video recording of the chef cooking, but there's a catch: the camera is broken. It records the ingredients on the counter and the chef's hand movements, but it fails to record the chef's nose sniffing the air or their eyes checking a timer.

In the real world, the chef might smell that the soup is getting too salty (a hidden piece of information) and decide not to add more salt. But in your broken video, it looks like the chef is ignoring the soup's state and acting randomly. If you try to guess the chef's "recipe" (their goals) just by watching the broken video, you might conclude the chef is bad at cooking or has a weird, inconsistent recipe.

This paper is about fixing that detective work. It introduces a new method called MP-IRL (Minimum-Perturbation Inverse Reinforcement Learning) to answer a simple question: "How much missing information do we need to assume exists to make the expert's actions look perfectly logical?"

Here is a breakdown of how it works, using everyday analogies:

1. The Problem: The "Broken Camera" Effect

In many fields, like healthcare, we try to learn from experts (like doctors) by looking at their past records.

The Scenario: A doctor treats a patient with high blood pressure with Medicine A, and a patient with low blood pressure with Medicine B.
The Missing Data: The hospital database only records the treatment, not the blood pressure reading at that exact moment.
The Mistake: If a computer tries to learn the doctor's "reward function" (what they are trying to achieve) from this incomplete data, it gets confused. It sees the doctor giving Medicine B to some patients and Medicine A to others, with no obvious pattern. It might think the doctor is making mistakes or that the reward function is messy.

2. The Solution: The "Invisible Variable"

Instead of just saying "the data is bad," this paper asks: "What is the smallest, simplest 'invisible variable' we can invent to make the doctor look like a genius again?"

Think of it like this:

You see a person running left at a crosswalk.
You also see the same person running right at a crosswalk later.
Without context, they look inconsistent.
The Paper's Approach: It invents a tiny, invisible "context tag" for each trip.
- Trip 1: [Context: "Red Light"] -> Action: Run Left.
- Trip 2: [Context: "Green Light"] -> Action: Run Right.
By adding just these tiny tags, the person's behavior suddenly makes perfect sense. The paper calculates how big these tags need to be. If the tags are huge, it means a lot of information is missing. If they are tiny, the missing information isn't a big deal.

3. How the Algorithm Works (The Two-Step Dance)

The authors built a two-step process to find these invisible tags:

Step 1: The "Best Guess" (Base IRL)
First, the computer tries to explain the expert's behavior using only the data it has (the broken video). It builds a "base reward" model. It's like saying, "Okay, based on what we see, here is the best recipe we can guess."
- Result: The computer realizes, "I'm still getting this wrong. The expert is doing things I can't explain."
Step 2: The "Minimal Fix" (MP-IRL)
Now, the computer freezes that base recipe. It asks: "What is the smallest amount of extra, invisible information I need to add to make the expert's actions fit this recipe perfectly?"
- It doesn't try to guess the exact missing data (like the exact blood pressure number).
- It just calculates the magnitude of the missingness. It's like measuring the "size of the hole" in the data without needing to fill the hole with the exact missing brick.

4. What They Found (The Experiments)

The team tested this on three types of "games":

Navigation Games: They created a maze where a robot had to choose left or right. Sometimes the choice depended on a hidden color the robot could see, but the researchers couldn't.
- Result: When the researchers hid the color, the standard method failed. MP-IRL successfully identified that a "hidden context" was needed and measured exactly how much "hiddenness" was required to fix the logic. It even figured out that if there were two hidden choices, the "missingness size" was bigger.
Cancer Treatment Simulator: They simulated a doctor treating a tumor.
- Result: When they hid important data (like tissue types), the "missingness size" grew larger. When they hid unimportant data, the size stayed small. This proves the method can tell the difference between "critical missing info" and "irrelevant missing info."
Real ICU Data (MIMIC-IV): They used real data from an Intensive Care Unit regarding blood pressure management.
- Result: Even with all the recorded data, the method found that a significant amount of "invisible context" (like a doctor's bedside intuition or unrecorded vitals) was likely needed to explain why doctors made certain choices. This suggests that if we try to train AI on this data without accounting for missing info, we might misunderstand the doctors' true goals.

The Bottom Line

This paper doesn't give you the missing data back. Instead, it gives you a ruler to measure the missingness.

It tells us: "If you want to trust the decisions made by experts in your dataset, you need to know how much of the picture is missing. If the 'missingness size' is huge, you can't trust the data to teach an AI how to act, because the AI will be learning from a broken picture. If the size is small, you can be more confident."

It's a tool for data quality control, helping researchers decide if their dataset is good enough to learn from or if they need to go back and record more details.

Technical Summary: Quantifying Potential Observation Missingness in Inverse Reinforcement Learning

Problem Statement
Inverse Reinforcement Learning (IRL) infers reward functions from expert demonstrations to model decision-making behavior. While variants of IRL have been developed to handle subjective beliefs, imperfect planning, and shifting goals, a critical issue often overlooked in real-world datasets (particularly in healthcare) is observation missingness. In many scenarios, the recorded data lacks observations that were available to the original decision-maker at the time of action.

When such information is missing, expert actions may appear suboptimal to a standard IRL algorithm, leading to misleading reward inferences. For instance, if a clinician's decision depends on an unrecorded biomarker, standard IRL might conclude the clinician is acting sub-optimally or that the reward function is inconsistent, rather than recognizing that the decision was optimal given the unobserved context. The paper posits that instead of attempting to impute missing data or recover the exact missing values, it is more valuable to quantify the minimal extent of missingness required to explain the observed behavior as optimal.

Methodology: Minimum-Perturbation IRL (MP-IRL)
The authors propose MP-IRL, a two-phase framework designed to find the smallest perturbation to recorded observations necessary to align expert actions with an optimal policy.

Problem Formulation:
- The setting assumes a behavioral dataset $D = \{x_{nt}, a_{nt}\}$ where $x_{nt}$ are recorded states and $a_{nt}$ are actions. The true state includes unrecorded components $\tilde{x}_{nt}$ .
- Rather than modeling the dynamics of missing data, the method introduces a static latent factor $z_n$ for each trajectory $n$ , representing the missing context.
- The objective is to minimize the magnitude of these latent factors $\{z_n\}$ subject to the constraint that the expert actions are near-optimal under a reward function $r_\theta(z_n, x_{nt}, a_{nt})$ .
- To ensure the magnitude of $z_n$ is meaningful, the authors parameterize the reward using kernel functions. The reward is a sum of a shared base reward $\bar{r}_\theta(x_{nt}, a_{nt})$ and a trajectory-specific perturbation term defined by kernel centers $\mu_k$ and coefficients $z_n$ . The size of the missingness is measured by the $\ell_1$ norm of the coefficients $\|z_n\|_1$ .
Algorithm (Two-Phase Optimization):
- Phase 1 (Base IRL): The algorithm performs standard IRL on the recorded data alone (setting all $z_n = 0$ ). This extracts the best possible base reward $\bar{r}_\theta$ that explains the behavior without any missing context.
- Phase 2 (MP-IRL): The base reward parameters are frozen. The algorithm then optimizes the critic network and the trajectory-specific latent coefficients $z_n$ (and optionally kernel centers $\mu$ ).
- Regularization: The objective function in Phase 2 includes sparsity penalties ( $\ell_1$ and $\ell_2$ norms on $z_n$ ) to enforce the "minimum-perturbation" principle. This ensures that the latent variables only explain the variance that the base reward cannot, preventing the model from overfitting or explaining away behavior through arbitrary changes to the shared reward structure.

Key Contributions

Conceptual: The paper introduces a new perspective on IRL missingness: quantifying the minimal unobserved context required to make observed actions appear optimal, rather than attempting to recover the missing data itself.
Technical: The development of MP-IRL, an algorithm that formulates this as a constrained optimization problem using kernel-based perturbations and a two-phase training procedure to separate shared reward structure from trajectory-specific missing context.
Empirical: Demonstration that the magnitude of the learned perturbations correlates with the amount of missing information and that the method can localize where missing context is critical (e.g., specific decision regions).

Experimental Results
The authors evaluate MP-IRL on three synthetic continuous navigation tasks, a cancer treatment simulator, and a real-world ICU hypotension management dataset (MIMIC-IV).

Navigation Tasks: In tasks where hidden binary variables determine actions in specific "decision regions," standard IRL fails to match expert actions within these regions (accuracy ~60%). MP-IRL restores accuracy to near 98-99%. Crucially, the learned perturbation size ( $\|z_n\|_1$ ) scales with the complexity of the missing information: tasks requiring one hidden bit yielded smaller perturbations than tasks requiring two independent hidden bits.
Cancer Simulator: The method was tested with varying degrees of feature masking (from "all features" to "time only"). As more features were hidden, standard IRL performance degraded (lower accuracy, higher Negative Log-Likelihood). MP-IRL maintained high accuracy, and the required perturbation size increased monotonically with the amount of missing information (e.g., from 0.54 to 1.65 as features were removed).
ICU Hypotension Task: On real clinical data, MP-IRL significantly improved action-matching accuracy over standard IRL (e.g., from ~55% to ~90%). The perturbation size remained stable when low-predictive features were added, indicating the metric reflects decision-relevant missingness rather than just the volume of observed data.

Significance and Claims
The paper claims that MP-IRL provides a diagnostic tool for behavioral datasets. By quantifying the minimal perturbation needed to explain expert behavior, researchers can assess whether apparent suboptimality in a dataset is due to poor decision-making or simply missing clinical context.

Practical Utility: If the potential missingness is substantial, it suggests that downstream tasks (like policy evaluation or offline RL) should be approached with caution or that data collection processes need improvement.
Interpretability: The method localizes missingness to specific regions of the state space (e.g., specific time steps in cancer treatment or specific decision boundaries in navigation), offering insights into where the recorded data is insufficient.
Limitations: The authors acknowledge that the kernel-based representation may struggle in very high-dimensional state spaces due to the data requirements for optimizing localized kernels.

The work does not claim to recover the exact missing values but rather to provide a quantitative bound on the extent of missingness required to maintain the assumption of expert optimality.