📊 statistics

Bayesian Inverse Transition Learning: Learning Dynamics From Near-Optimal Trajectories

This paper proposes Bayesian Inverse Transition Learning, a novel constraint-based method that leverages the near-optimality of expert trajectories to estimate transition dynamics and improve decision-making in offline model-based reinforcement learning, particularly in data-scarce healthcare scenarios like ICU hypotension management.

Original authors: Leo Benac, Abhishek Sharma, Sonali Parbhoo, Finale Doshi-Velez

Published 2026-04-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Leo Benac, Abhishek Sharma, Sonali Parbhoo, Finale Doshi-Velez

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to navigate a complex maze or manage a patient's health, but you don't have a manual. You only have a video recording of an expert doing the job. The problem is, the expert only shows you a few specific paths through the maze, and they never show you what happens if you take a wrong turn.

This is the challenge the paper tackles: How do you learn the "rules of the world" (the physics or biology) when you only have a limited, imperfect view of an expert's success?

Here is a simple breakdown of their solution, Bayesian Inverse Transition Learning (ITL and BITL), using everyday analogies.

1. The Problem: The "Blind Map"

In traditional learning, you might try to guess the rules of the world just by counting how often things happen. If you see a doctor give a drug and the patient gets better 10 times, you assume the drug causes the improvement.

But this is dangerous if the data is sparse.

The Analogy: Imagine you are trying to learn the rules of a new board game by watching a grandmaster play only three games. You see them move a piece to the top-left corner and win. You might guess, "Ah, moving to the top-left always wins!" But you don't know what happens if they moved to the top-right, because the grandmaster never did that.
The Flaw: Standard methods (like Maximum Likelihood) would just say, "We have no data on the top-right, so we have no idea what happens there." This leads to bad guesses.

2. The Insight: The Expert is "Near-Optimal"

The authors realized they could use a powerful clue: The expert is good. They aren't perfect, but they are very close to the best possible player.

The Analogy: If a grandmaster chooses to move their piece to the top-left instead of the top-right, it implies that the top-left path is at least as good as the top-right path. Even if we don't know the exact score of the top-right path, we know it's not better than the one they chose.
The Paper's Move: Instead of just counting what happened, they use the expert's choices to set hard rules (constraints). They say: "The path the expert took must be better than the paths they ignored."

3. The Solution: "Inverse Transition Learning" (ITL)

The authors created a new method called Inverse Transition Learning (ITL). Think of this as a "Logic Puzzle Solver."

How it works: Instead of guessing the rules and hoping they fit the data, ITL starts with the data and asks, "What set of rules would make the expert's choices the best possible choices?"
The "Hard Constraints": They force the computer to find a model of the world where:
1. The actions the expert took are definitely good.
2. The actions the expert didn't take are definitely worse (or at least not better).
The Benefit: This is like solving a Sudoku puzzle. You don't guess randomly; you use the numbers already on the board to eliminate impossible options. This prevents the computer from getting stuck in "local optima" (bad guesses that look okay but are wrong) and makes the learning much faster and more reliable.

4. The "Bayesian" Twist: Knowing What You Don't Know

The paper also introduces BITL (Bayesian Inverse Transition Learning).

The Analogy: ITL gives you one single "best guess" map of the world. But what if you want to know how confident you should be in that map?
How it works: BITL doesn't just give you one map; it gives you a cloud of possible maps. Some maps look very similar to the expert's path, others look a bit different but still fit the rules.
Why it matters: This allows the system to say, "I am very sure about this part of the maze, but I am totally guessing about that dark corner."
The Superpower: The paper shows that this "cloud of uncertainty" can predict when the robot will fail. If the robot tries to move to a place where the "cloud" is very wide (high uncertainty), the system knows, "Hey, we haven't seen this before; be careful." This helps in deciding when to transfer skills to a new situation (like a new patient or a new maze).

5. Real-World Testing: The ICU

The authors tested this on two types of environments:

Synthetic Mazes: Simple grid worlds and random mazes.
Real Healthcare: Managing low blood pressure (hypotension) in ICU patients using real data from the MIMIC-IV database.

The Results:

Speed: Their method was dramatically faster than previous methods (seconds vs. minutes/hours).
Accuracy: In the ICU scenario, their method learned the "rules" of patient recovery much better than standard methods.
Safety: Because they used "hard constraints," their method never suggested a treatment that the expert would have clearly avoided. It stayed within the "safe zone" of expert behavior.
Transfer: When they changed the goal (e.g., prioritizing a different health metric), their method adapted better because it understood the underlying "physics" of the patient, not just the specific goal.

Summary

The paper presents a smarter way to learn from experts when data is scarce. Instead of just copying what the expert did, it asks, "What must the world look like for the expert to have made those choices?"

By turning the expert's choices into strict logical rules, they can build a much more accurate and reliable model of how the world works, even with very little data. It's like deducing the rules of a game not by reading the manual, but by watching a pro play and realizing, "If they didn't do X, X must be a bad move," and using that logic to fill in the blanks.

1. Problem Statement

The paper addresses the challenge of Offline Model-Based Reinforcement Learning (MBRL) in settings where the transition dynamics ( $T^*$ ) of an environment are unknown, but a batch of data generated by a near-optimal expert is available.

Context: In fields like healthcare (e.g., ICU management) and education, data is often observational and limited. The expert (e.g., a doctor) is not perfectly optimal but follows a policy that is "near-optimal" ( $\epsilon$ -optimal).
Challenge: Traditional Maximum Likelihood Estimation (MLE) fails to leverage the expert's near-optimality, leading to poor dynamics estimates when data coverage is low. Existing Inverse Reinforcement Learning (IRL) methods (like Maximum Causal Entropy, MCE) often rely on soft constraints, gradient-based optimization (prone to local optima), and lack guarantees on the learned dynamics.
Goal: To estimate the true transition dynamics $T^*$ and a posterior distribution over it, leveraging the fact that the expert avoids suboptimal actions, while ensuring the learned model respects hard constraints derived from expert behavior.

2. Methodology

The authors propose two related methods: Inverse Transition Learning (ITL) for point estimates and Bayesian Inverse Transition Learning (BITL) for posterior distributions.

A. Core Concept: $\epsilon$ -Optimality and Constraints

The method assumes the expert policy $\pi_\epsilon$ selects actions from an $\epsilon$ -ball of near-optimal actions. This leads to two hard constraints on the learned dynamics $T$ :

Separation Constraint: For any state $s$ , the $Q$ -value of any valid (observed/near-optimal) action must be at least $\epsilon$ higher than any invalid (unobserved/suboptimal) action.
$Q^*(s, a_{valid}; T) - Q^*(s, a_{invalid}; T) \geq \epsilon$
Closeness Constraint: For any pair of valid actions, their $Q$ -values must be within $\epsilon$ of each other.
$|Q^*(s, a_1; T) - Q^*(s, a_2; T)| \leq \epsilon$

These constraints ensure the learned dynamics distinguish between good and bad actions while acknowledging that multiple good actions may exist (common in clinical settings where different drugs have similar efficacy).

B. Inverse Transition Learning (ITL) - Point Estimate

Formulation: ITL formulates the estimation of $T$ as a convex quadratic optimization problem.
Objective: Minimize the distance between the learned $T$ and the standard MLE ( $T^{MLE}$ ) subject to the hard constraints defined above.
Linearization: To maintain convexity, the constraints are linearized using a fixed reference policy and transition dynamics (updated iteratively).
Solver: The problem is solved efficiently using CVXPY (a convex optimization library), avoiding the local optima issues of gradient-based methods like MCE.
Result: A deterministic point estimate $\hat{T}^*$ that strictly satisfies the $\epsilon$ -ball property.

C. Bayesian Inverse Transition Learning (BITL) - Posterior Distribution

Motivation: In offline settings with limited data, uncertainty remains. BITL aims to learn a posterior distribution $P(T^* | D)$ that incorporates the expert constraints.
Sampling Technique: Since the constraints create a non-convex feasible region, standard rejection sampling is inefficient. The authors employ Hamiltonian Monte Carlo (HMC) with Reflection.
- The algorithm maps the constrained simplex space of transition probabilities to an unconstrained space.
- During the leapfrog integration steps, if a trajectory hits a constraint boundary, the momentum is reflected off the boundary rather than the sample being rejected.
Warm Start: The HMC chain is initialized with the point estimate from ITL to ensure feasibility.
Benefit: This yields a calibrated distribution over dynamics, allowing for uncertainty quantification and better transfer learning predictions.

3. Key Contributions

Novel Constraint-Based Framework: Introduction of ITL, which uses hard constraints derived from near-optimal expert behavior to estimate dynamics, contrasting with soft-constraint IRL methods.
Efficient Optimization: The method avoids gradient-based alternating optimization (which is computationally expensive and prone to local optima) by using a convex quadratic program, resulting in significantly faster convergence.
Bayesian Extension (BITL): The first method to infer a posterior distribution over transition dynamics using expert demonstrations, utilizing HMC with reflection to handle complex constraints.
Transfer Prediction: The Bayesian posterior allows the system to predict when a policy learned under one reward function will successfully transfer to a new reward function (e.g., different clinical goals) by analyzing the regret of the posterior samples.
Real-World Application: Demonstration of the method in a high-stakes, data-limited healthcare scenario (ICU hypotension management) using the MIMIC-IV dataset.

4. Experimental Results

The authors evaluated their methods on:

Synthetic Environments: Gridworld (25 states) and 10 Randomworlds (15 states).
Real-World Data: ICU patient management for hypotension using MIMIC-IV.

Key Findings:

Performance: ITL and BITL consistently outperformed baselines (MLE, MCE, and non-expert Bayesian methods) across all metrics, including Best Action Matching, $\epsilon$ -ball Matching, and Normalized Value.
Robustness: In Gridworld, while MCE sometimes matched ITL in average performance, ITL achieved significantly better worst-case performance (lower CVaR) and zero constraint violations, whereas MCE frequently violated constraints due to local optima.
Efficiency: ITL is orders of magnitude faster than MCE.
- Gridworld: ITL took ~0.3s vs. MCE's ~117s.
- Randomworld: ITL took ~0.65s vs. MCE's ~36s.
Transfer Learning: BITL successfully predicted transfer success. Tasks with lower Bayesian Regret (calculated from the posterior) correlated with higher performance in transfer tasks (different reward functions).
Healthcare Application: In the ICU setting, ITL/BITL significantly outperformed MLE and MCE. Notably, MCE often collapsed probability onto a single unrealistic state, whereas ITL provided a more realistic distribution of outcomes that respected physiological constraints.

5. Significance and Impact

Safety and Reliability: By enforcing hard constraints, the method guarantees that the learned dynamics respect the expert's behavior, which is critical in safety-critical domains like healthcare where violating expert knowledge could lead to harmful policies.
Data Efficiency: The approach effectively utilizes sparse, offline data by leveraging the structural information of near-optimality, making it suitable for scenarios where collecting new data is impossible or dangerous.
Interpretability: The method provides a transparent mechanism (via the $\epsilon$ parameter) to control the trade-off between strict adherence to expert data and flexibility, and the resulting dynamics are more interpretable than black-box neural network models.
Clinical Relevance: The ability to handle "stochastic-policy states" (where multiple treatments are equally valid) makes the method uniquely suited for clinical decision-making, where doctors often have multiple reasonable options rather than a single "correct" action.

In summary, this paper presents a robust, fast, and theoretically grounded approach to learning transition dynamics from limited expert data, bridging the gap between theoretical RL constraints and practical, high-stakes applications in healthcare.