🤖 machine learning

Distributional Inverse Reinforcement Learning

This paper proposes a novel distributional framework for offline Inverse Reinforcement Learning that jointly models reward function uncertainty and full return distributions by minimizing first-order stochastic dominance violations, thereby enabling the recovery of expressive reward representations and risk-aware policies with proven convergence guarantees.

Original authors: Feiyang Wu, Ye Zhao, Anqi Wu

Published 2026-04-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Feiyang Wu, Ye Zhao, Anqi Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to walk, or a computer how to play a video game, but you don't know the rules or the points system. You only have a video of an expert doing it perfectly. This is the core problem of Inverse Reinforcement Learning (IRL): figuring out what the expert is trying to achieve just by watching how they do it.

For a long time, AI researchers assumed the "points" (rewards) were simple and fixed. If you jumped, you got exactly 10 points. Every time.

The Problem: Life is Messy
In the real world, rewards are rarely that predictable.

The Robot Arm: If a robot tries to pick up a fragile egg, sometimes it succeeds (great reward), sometimes it drops it (bad reward), and sometimes it squeezes it just enough to crack it (medium reward). The outcome is a gamble.
The Mouse: In a neuroscience experiment, a mouse's brain releases dopamine (a "reward chemical") differently every time it does the same action. It's not a fixed number; it's a fluctuating wave.

Old AI methods tried to guess the average reward. They asked, "On average, how many points did the expert get?" But this misses the whole story. It doesn't tell you if the expert was playing it safe to avoid a huge failure, or if they were taking wild risks for a huge win. It's like judging a poker player only by their average winnings, ignoring whether they are a cautious saver or a reckless gambler.

The Solution: DistIRL (Distributional Inverse Reinforcement Learning)
The authors of this paper propose a new method called DistIRL. Instead of guessing a single number for the reward, DistIRL guesses the entire shape of the reward.

Think of it like this:

Old Method (The Average): "The expert usually gets about 50 points."
DistIRL (The Full Picture): "The expert usually gets 50 points, but sometimes they get 100, sometimes they get 0, and they really hate getting negative points. Their strategy is to avoid the 0s at all costs."

How It Works: The "FSD" Analogy
To learn this full picture, the paper uses a concept called First-Order Stochastic Dominance (FSD). Let's use a metaphor of two different weather forecasts:

Forecast A (The Expert): "It will rain, but mostly light drizzle. Maybe a heavy storm once in a while, but mostly manageable."
Forecast B (Your Robot): "It might be sunny, or it might be a hurricane."

Even if both forecasts have the same average rainfall, Forecast A is "better" (it dominates) because it guarantees you won't get soaked in a hurricane.

DistIRL looks at the expert's behavior and says: "My robot's strategy must be better than or equal to the expert's strategy across every possible outcome, not just the average." It forces the robot to learn the full distribution of risks and rewards, ensuring it doesn't accidentally learn a strategy that is risky in ways the expert wasn't.

Why This Matters: The "Risk-Aware" Agent
Because DistIRL understands the full distribution, it can learn risk-aware policies.

If the expert is a risk-averse driver (always staying in the slow lane to avoid accidents), DistIRL learns that the "reward" for speeding is actually a huge risk of a crash, even if the average speed is high.
If the expert is a risk-seeking investor, DistIRL learns that they are willing to accept huge losses for a chance at a massive win.

Real-World Tests
The team tested this in three ways:

Gridworld (A simple maze): They showed that DistIRL could figure out that an expert was avoiding a "risky" high-reward spot because it sometimes gave zero reward, whereas old methods thought the spot was just "okay."
Mouse Brains (Neuroscience): They analyzed real data from mice. The "reward" in a mouse's brain (dopamine) is messy and variable. DistIRL successfully reconstructed the shape of these dopamine spikes, matching the real biological data much better than old methods. It proved that behavior alone can reveal the complex, fluctuating nature of internal motivation.
Robot Control (MuJoCo): In complex robot simulations where falling over is a disaster, DistIRL learned to make robots move safely and efficiently, matching the performance of the "expert" robots better than any previous method.

The Bottom Line
This paper is like upgrading from a black-and-white TV to a 4K HDR screen. Old methods saw the "average" of the expert's world. DistIRL sees the color, the depth, and the shadows. It allows AI to understand not just what an expert wants, but how they feel about the uncertainty of the future, leading to smarter, safer, and more human-like AI.

1. Problem Statement

Inverse Reinforcement Learning (IRL) aims to infer an expert's underlying reward function and policy from observed trajectories. While traditional IRL methods (e.g., MaxEntIRL) have been successful, they rely on a critical limitation: they assume the reward function is deterministic, recovering only a point estimate $r(s, a) \in \mathbb{R}$ .

In many real-world scenarios, reward signals are inherently stochastic:

Robotics: Contact uncertainty with deformable objects leads to variable rewards for identical state-action pairs.
Neuroscience: Dopamine signals (often used as reward proxies) exhibit significant trial-to-trial variability.
Risk Sensitivity: A deterministic reward model only captures the mean of the return distribution. It fails to capture higher-order moments (variance, skewness), making the learned policy insensitive to risk preferences (e.g., risk-averse vs. risk-seeking behavior).

Existing Bayesian IRL (BIRL) methods attempt to model uncertainty but typically infer a posterior over parameters of a deterministic reward function and still optimize for expected return, failing to recover the full shape of the reward distribution or the resulting return distribution.

The Core Challenge: How to learn the full distribution of the reward function and the full distribution of returns directly from offline expert demonstrations, enabling risk-aware policy learning.

2. Methodology: DistIRL Framework

The authors propose DistIRL (Distributional Inverse Reinforcement Learning), a framework that jointly models reward uncertainty and return distributions.

A. First-Order Stochastic Dominance (FSD) for Reward Learning

Instead of matching expected returns (as in MaxEntIRL), DistIRL aims to align the entire return distribution of the agent with that of the expert.

Objective: The algorithm minimizes violations of First-Order Stochastic Dominance (FSD). If the expert's return distribution $Z_E$ stochastically dominates the agent's $Z_\pi$ ( $Z_E \succeq_{FSD} Z_\pi$ ), the agent is considered to be performing at least as well as the expert across all risk profiles.
Loss Function: The FSD violation is formulated as an energy function:
$L_{FSD}(\pi, r) = \int_{-\infty}^{\infty} [F_{Z_E}(z) - F_{Z_\pi}(z)]_+ dz$
where $[x]_+ = \max(x, 0)$ . This integral measures the area where the expert's Cumulative Distribution Function (CDF) is above the agent's, indicating a dominance violation.
Variational Inference: To learn the reward distribution $q_\phi(r|s, a)$ , the authors use an Energy-Based Model (EBM) formulation combined with Variational Inference. They approximate the posterior of the reward distribution by minimizing an Evidence Lower Bound (ELBO) that includes the expected FSD violation and a KL-divergence regularization against a prior.
Quantile Approximation: To make the FSD loss tractable, they utilize the identity relating CDF integrals to quantile functions:
$\int_{-\infty}^{\infty} [F_{Z_E}(z) - F_{Z_\pi}(z)]_+ dz = \int_{0}^{1} [F^{-1}_{Z_\pi}(v) - F^{-1}_{Z_E}(v)]_+ dv$
This allows the use of Monte Carlo sampling and empirical quantiles to estimate the loss.

B. Risk-Aware Policy Learning via Distortion Risk Measures (DRMs)

Directly optimizing the FSD objective for the policy is intractable because the indicator function defining the violation regions is non-differentiable and depends on the unknown expert distribution.

Surrogate Objective: The authors replace the intractable indicator function with a Distortion Risk Measure (DRM). A DRM, $M_\xi(X)$ , aggregates the quantiles of a distribution weighted by a distortion function $\xi$ .
$M_\xi(X) = \int_0^1 F^{-1}_X(v) d\tilde{\xi}(v)$
Theoretical Justification: They prove that if a policy maximizes the DRM for all possible distortion functions, it satisfies the FSD condition. In practice, they select a specific DRM (e.g., CVaR) that aligns with the expert's risk profile to optimize the policy.
Algorithm: The policy is updated to maximize the DRM of the return distribution plus an entropy term, effectively learning a policy that is aware of the full distribution of outcomes, not just the mean.

C. Algorithm Overview

The algorithm alternates between three steps (Algorithm 1):

Critic Update: Uses Quantile Regression (QR) with Quantile Huber loss to estimate the return distribution $Z_\pi$ for the current policy.
Policy Update: Updates the policy parameters to maximize the chosen Distortion Risk Measure (DRM) of the return distribution.
Reward Distribution Update: Updates the parameters of the reward distribution $q_\phi$ to minimize the FSD violation loss between the expert and agent return distributions.

3. Key Contributions

Reward Distribution Learning: DistIRL is the first framework to learn the full distribution of the reward function (including higher-order moments like variance and skewness) in a principled offline IRL setting, moving beyond point estimates or parameter posteriors.
Distribution-Aware Policy Learning: By integrating FSD and DRMs, the method recovers policies that are explicitly risk-aware, capable of capturing risk-averse or risk-seeking behaviors inherent in the expert data.
Theoretical Analysis: The authors provide a convergence analysis showing the algorithm converges to a first-order stationary point with an iteration complexity of $O(\epsilon^{-2})$ .
Empirical Validation: The method is validated on:
- Synthetic Gridworld: Successfully recovering reward variance and risk-averse behavior.
- Neuroscience Data: First-time application of IRL to mouse spontaneous behavior, successfully inferring reward distributions that match the statistical properties of dopamine fluctuations (including skewness).
- MuJoCo Control: State-of-the-art performance on risk-sensitive D4RL benchmarks (HalfCheetah, Hopper, Walker2D) with stochastic safety penalties.

4. Experimental Results

Gridworld: DistIRL correctly identified the mean and variance of stochastic rewards, whereas Bayesian IRL (BIRL) recovered the mean but failed to capture variance, leading to spurious high estimates in low-reward areas.
Mouse Behavior (Neuroscience):
- Using a skew-normal distribution, DistIRL achieved the highest correlation (approx. 0.3) between inferred reward means and actual dopamine signals.
- Crucially, DistIRL recovered the full shape (skewness and multimodality) of the dopamine distribution, whereas deterministic models and BIRL failed to capture these higher-order statistics.
MuJoCo (Risk-Sensitive D4RL):
- DistIRL significantly outperformed baselines (Offline ML-IRL, ValueDICE, Behavior Cloning) in environments with stochastic safety penalties (e.g., high-speed crashes).
- It achieved returns close to the expert (e.g., 3469 vs 3540 on HalfCheetah) while baselines struggled due to distributional mismatch or risk-neutrality.
- Return Distribution Matching: Visualizations showed DistIRL's inferred return distribution closely matched the expert's, including tail behavior, while BIRL deviated significantly.

5. Significance and Impact

Bridging IRL and Distributional RL: DistIRL unifies the concepts of Inverse Reinforcement Learning and Distributional Reinforcement Learning, allowing for the inference of complex, stochastic reward structures.
Neuroscientific Insight: The ability to decode the full distribution of internal motivational states (like dopamine variability) from behavior alone opens new avenues for understanding animal and human decision-making under uncertainty.
Safety and Robustness: In robotics and autonomous systems, where safety is paramount, learning risk-averse policies that account for reward uncertainty is critical. DistIRL provides a mechanism to learn these policies directly from demonstrations without requiring explicit risk modeling by the designer.
Generalizability: The framework is flexible regarding the choice of prior and distortion function, making it adaptable to various domains where reward signals are noisy or risk-sensitive.

In summary, DistIRL represents a significant leap forward in IRL by acknowledging that "rewards are distributions, not numbers," thereby enabling more robust, expressive, and biologically plausible models of expert behavior.