📈 economics

A Lecture Note on Offline RL and IRL, Part II: Foundations of Inverse Reinforcement Learning and Dynamic Discrete Choice Models

This lecture note establishes the theoretical equivalence between structural econometric Dynamic Discrete Choice models and entropy-regularized Inverse Reinforcement Learning, systematically comparing classical identification and computational methods with modern machine learning approaches to clarify their respective objectives, limitations, and identification guarantees for offline reward recovery.

Original authors: Enoch Hyunwook Kang

Published 2026-06-01

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Enoch Hyunwook Kang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The Mystery of the "Why"

Imagine you are a detective trying to figure out why a master chef cooks a specific dish the way they do.

Forward Reinforcement Learning (The Standard Way): You are given the recipe (the reward) and the ingredients. Your job is to learn how to cook the dish perfectly.
Inverse Reinforcement Learning (IRL) & Dynamic Discrete Choice (DDC): You are only given a video of the master chef cooking. You don't have the recipe. Your job is to watch the video and figure out the hidden recipe (the reward) that explains why they made those specific choices.

This lecture note is a guidebook for two different groups of detectives who have been solving this same mystery for decades, but they speak different languages and use different tools. The author, Enoch Kang, shows that they are actually solving the exact same puzzle and then introduces a new, unified tool to solve it better.

Part 1: Two Languages, One Puzzle

The paper starts by proving that two seemingly different fields are actually looking at the same thing:

Economists (DDC): They study how people make choices (like picking a job or a car). They assume people have a hidden "utility" (happiness score) plus some random noise (a bad day, a sudden craving). They use math to figure out the hidden utility.
AI Researchers (IRL): They study robots or agents. They assume the agent is trying to maximize a reward but also likes to be "random" (exploratory) to avoid getting stuck. This randomness mathematically looks exactly like the economist's "noise."

The Analogy: Imagine two people describing a cloud. One says, "It's a fluffy white shape." The other says, "It's a water vapor formation." They are describing the same object with different words. This paper proves that the "fluffy shape" (Econ) and the "water vapor" (AI) are mathematically identical.

Part 2: The "Anchor" Problem (The Missing Piece)

Here is the tricky part: If you only watch the chef, you can't tell if they added salt because they love salt, or because they wanted to balance the acidity, or because they were trying to hide a mistake. There are infinite ways to explain the same behavior.

The Problem: You can't uniquely identify the "true" reward just by watching behavior. You can only identify differences between choices (e.g., "The chef prefers pasta over rice"), but not the absolute value (e.g., "How much does the chef love pasta?").
The Solution (The Anchor): To solve this, the paper suggests picking one specific action at every step and declaring its value is known.
- Analogy: Imagine the chef always puts a specific amount of salt in the soup when they are making "Recipe A." If we know that "Recipe A" always gets exactly 1 gram of salt, we can use that as a ruler (an Anchor) to measure how much salt they put in "Recipe B."
- In the paper, this is called the Anchor-Action Assumption. It fixes the scale so the math works.

Part 3: The Old Tools (Why They Were Hard)

The paper reviews the old ways detectives tried to solve this, pointing out their flaws:

The Nested Loop (Rust's Method):
- How it worked: Guess a recipe, simulate the chef cooking to see what they would do, compare to the video, and repeat.
- The Flaw: It's like trying to solve a maze by walking through it, then walking back, then walking through again. It's incredibly slow and computationally expensive, especially if the maze is huge (high-dimensional).
The Conditional Choice (Hotz-Miller):
- How it worked: Instead of guessing the recipe, they guessed the chef's next move probabilities and worked backward.
- The Flaw: To do this, you need to know exactly how the world changes (the transition model). If you don't know how the kitchen moves (e.g., how the stove heats up), this method fails. It requires estimating a massive map of the world, which is statistically impossible in complex environments.
The "Deadly Triad" (Temporal Difference):
- How it worked: Trying to learn directly from the video clips without simulating the whole world.
- The Flaw: When you combine approximation (guessing), bootstrapping (using your own guess to update your guess), and off-policy data (learning from a different chef than the one you are trying to mimic), the math often explodes. The numbers go to infinity, and the system crashes.

Part 4: The Modern AI Tricks (Adversarial & Matching)

The paper then looks at modern AI methods like AIRL and GAIL.

The Idea: Use a "Discriminator" (a referee) to play a game. The referee tries to tell if a move came from the expert or a student. The student tries to fool the referee.
The Limit: The paper argues that while these methods are cool, they often fail to actually find the true reward. They might just find a way to mimic the behavior without understanding the "why." They often rely on assumptions (like the world being deterministic) that aren't true in real life. If the world is random (stochastic), these methods get confused about what part of the behavior is the reward and what part is just luck.

Part 5: The New Solution (GLADIUS)

Finally, the paper introduces a new method called GLADIUS (Gradient-based Learning with Ascent–Descent for Inverse Utility learning from Samples).

How it works (The Analogy):
Imagine you are trying to find the perfect temperature for a shower.

The Likelihood Loss: You look at the video of the expert. You adjust the temperature until the expert's choices match the video. This gets you the relative preferences (Hot vs. Cold).
The Anchor Loss: You use the "Anchor" (the known salt amount) to fix the absolute scale.
The Bias Correction (The Magic Trick):
- The Problem: If you just look at one video clip, you might get lucky or unlucky with the next state (e.g., the water pressure fluctuates). If you try to calculate the "error" based on just one clip, you get a biased result (the "Double Sampling" problem).
- The Fix: GLADIUS uses a second "helper" network (called $\zeta$ ). This helper acts like a statistician. It looks at all the data and predicts the average outcome of the next step, effectively canceling out the luck/unluckiness of any single clip.
- It does this by playing a game: The main network tries to minimize the error, while the helper tries to predict the average. They take turns updating each other.

Why it's better:

No Map Needed: It doesn't need to know the transition model (how the world moves). It learns directly from the video clips.
No Nested Loops: It doesn't need to simulate the whole future. It solves everything in one shot using gradients (mathematical slopes).
Stable: It avoids the "Deadly Triad" that crashes other methods.

Summary

The paper is a bridge. It connects the rigorous math of economists with the powerful tools of AI. It shows that while we have many ways to guess a reward function from behavior, most of them are either too slow, too unstable, or require impossible assumptions.

The proposed solution, GLADIUS, is a new way to solve the puzzle. It uses a "ruler" (the anchor action) to set the scale and a "statistician helper" (the bias correction) to ignore the noise in the data. This allows us to recover the true "recipe" (reward function) directly from the video, without needing to simulate the world or know the rules of the game beforehand.

Technical Summary: Foundations of Inverse Reinforcement Learning and Dynamic Discrete Choice Models

1. Problem Definition

The paper addresses the Inverse Reinforcement Learning (IRL) problem in the offline setting: given a fixed dataset of state-action-next-state tuples generated by an expert policy $\pi^*$ , can we recover the reward function $r$ that the expert was optimizing?

This problem sits at the intersection of two distinct literatures:

Structural Econometrics (Dynamic Discrete Choice - DDC): Models agents maximizing utility plus idiosyncratic Gumbel-distributed shocks.
Machine Learning (Entropy-Regularized IRL): Models agents maximizing expected return plus an entropy bonus (Maximum Entropy IRL).

The core challenge is identification: without additional structure, the reward function is not uniquely recoverable from behavior alone due to potential-based shaping invariance (Ng et al., 1999). Furthermore, existing methods face a "trilemma" of trade-offs involving scalability, the need for transition kernel estimation, and stability (the "deadly triad").

2. Methodological Foundations

2.1 Equivalence of DDC and MaxEnt-IRL

The paper first establishes a rigorous equivalence between the two communities.

DDC Formulation: Agents choose actions to maximize $r(s,a) + \epsilon_{t,a}$ , where $\epsilon$ follows a Type-1 Extreme Value (Gumbel) distribution.
MaxEnt-IRL Formulation: Agents maximize expected return plus an entropy term $\lambda H(\pi)$ .
Result: Under a mean-zero normalization for the Gumbel shocks ( $\delta = -\gamma_E$ ) and setting the entropy coefficient $\lambda=1$ , both formulations yield the exact same softmax policy and soft Bellman equation:
$Q^*(s, a) = r(s, a) + \beta \mathbb{E}_{s' \sim P} \left[ \log \sum_{a'} \exp(Q^*(s', a')) \right]$
$\pi^*(a|s) = \frac{\exp(Q^*(s, a))}{\sum_{a'} \exp(Q^*(s, a'))}$
This proves that the probabilistic models are statistically indistinguishable from offline observations.

2.2 Identification Limits

The paper analyzes what can be recovered from behavior alone:

Non-Identification: Without further assumptions, $r$ is only identifiable up to a state-dependent potential function $\Phi(s)$ . Specifically, $r$ and $\tilde{r}(s,a) = r(s,a) + \Phi(s) - \beta \mathbb{E}[\Phi(s')]$ induce the same optimal policy.
The Anchor-Action Assumption (Magnac & Thesmar, 2002): To resolve this, the paper adopts the classical econometric assumption that for every state $s$ , there exists a known "anchor" action $a_s$ with a known reward $r(s, a_s) = 0$ (or known constant). This assumption fixes the potential and allows for the unique identification of $Q^*$ and $r$ on the expert's support.

3. Review of Existing Approaches and Their Limitations

The paper surveys classical and modern methods, highlighting their specific failure modes:

3.1 Classical DDC Methods

Rust's Nested Fixed-Point (NFXP): Solves the Bellman equation in an inner loop while maximizing likelihood in an outer loop.
- Limit: Suffers from the curse of dimensionality in the state space $S$ and requires explicit knowledge or estimation of the transition kernel $P$ .
Hotz-Miller / Conditional Choice Probabilities (CCP): Inverts the softmax to find $Q$ $Q$ differences and solves a linear fixed-point equation for the value function.
- Limit: Requires estimating the transition kernel $P$ (statistical curse of dimensionality) and suffers from error propagation in forward simulation.
Temporal-Difference (TD) Approaches (Adusumilli & Eckardt, 2019):
- Linear Semi-Gradient TD: Avoids explicit $P$ but suffers from the deadly triad (function approximation + bootstrapping + off-policy data), leading to divergence.
- Approximate Value Iteration (AVI): More stable but converges to a projected fixed point, which may differ from the true $Q^*$ if the true value function is not realizable in the function class.
- Note: "Target networks" (common in deep RL) stabilize the trajectory but do not change the destination; they still converge to the projected fixed point, not the true value.

3.2 Modern IRL Methods

Adversarial IRL (AIRL): Uses a discriminator to separate reward from shaping potential.
- Limit: Identifies the soft advantage $A^*(s,a) = \log \pi^*(a|s)$ , not the reward directly. Reward recovery requires restrictive assumptions (deterministic dynamics, state-only rewards, decomposable transition graphs) that are often violated in practice.
Occupancy Matching (GAIL): Minimizes divergence between expert and policy occupancy measures.
- Limit: It is an imitation algorithm, not a reward recovery algorithm. It does not enforce pointwise Bellman consistency, making reward extraction impossible without further assumptions.
IQ-Learn: Parametrizes $Q$ $Q$ directly and defines a reward via the Bellman equation.
- Limit: The objective is invariant under state-only shifts; it does not identify a unique reward without an external normalization (like the anchor action).
Offline ML-IRL (Zeng et al., 2023): A neural re-skinning of Hotz-Miller with conservatism penalties.
- Limit: Inherits the need for transition kernel estimation and nested optimization loops.

4. Proposed Method: The ERM Framework (GLADIUS)

The paper proposes a unified Empirical Risk Minimization (ERM) framework that synthesizes the identification power of DDC with the scalability of modern ML, avoiding the pitfalls of both.

4.1 Core Insight

Instead of parameterizing $r$ and solving a nested forward problem, the method parametrizes $Q$ directly. The identification conditions are translated into two loss functions:

Negative Log-Likelihood (NLL): Enforces that the softmax of $Q$ matches the expert policy $\pi^*$ . This identifies within-state differences of $Q$ .
Anchor Bellman Loss: Enforces the Bellman equation only at the anchor action $a_s$ where the reward $r(s, a_s)$ is known. This fixes the state-wise level of $Q$ .

4.2 The Double-Sampling Bias and Correction

A naive empirical Bellman loss using squared TD errors is biased due to the double-sampling problem (variance of the next state does not vanish with more data).

Correction: The paper derives a minimax formulation that subtracts the conditional variance term.
$\min_Q \max_\zeta \mathbb{E} \left[ \text{NLL} + \mathbb{1}_{a=a_s} \left( (\text{Sampled Bellman})^2 - \beta^2 (V_Q(s') - \zeta(s,a))^2 \right) \right]$
Here, $\zeta(s,a)$ acts as an auxiliary function learning the conditional mean $\mathbb{E}[V_Q(s') | s, a]$ . This correction removes the need to estimate the transition kernel $P$ explicitly.

4.3 Algorithm: GLADIUS

The GLADIUS (Gradient-based Learning with Ascent–Descent for Inverse Utility learning from Samples) algorithm alternates between:

Regression Step (Ascent/Descent): Update $\zeta$ to minimize the squared error against $V_Q(s')$ .
Optimization Step (Descent): Update $Q$ to minimize the NLL plus the bias-corrected anchor Bellman residual.

4.4 Convergence Guarantees

The paper establishes convergence under Polyak-Lojasiewicz (PL) geometry assumptions rather than convexity.

Assumptions: Realizability of $Q^*$ and $\zeta^*$ in the function classes, and empirical-output Jacobian conditioning (similar to Neural Tangent Kernel conditions).
Result: The algorithm converges to the global minimum of the empirical risk with a rate of $O(1/T)$ for optimization and $O(1/\sqrt{N})$ for statistical error.

5. Key Contributions and Significance

Unification: Proves the formal equivalence between the DDC and MaxEnt-IRL formulations, bridging the gap between econometrics and machine learning.
Identification Clarity: Clarifies that modern adversarial methods (AIRL) do not inherently solve the reward identification problem without strong, often unrealistic, structural assumptions. It reaffirms the anchor-action assumption as the robust path to identifying state-action rewards.
Transition-Free Estimation: Introduces a bias-corrected minimax objective that recovers $Q^*$ and $r$ without estimating the transition kernel $P$ , overcoming the statistical curse of dimensionality inherent in CCP and NFXP.
Stability: Avoids the "deadly triad" divergence of off-policy TD methods by formulating the problem as a direct, bias-corrected empirical risk minimization over $Q$ .
Algorithmic Solution: Presents GLADIUS, a gradient-based, one-shot estimator that avoids the nested optimization loops of classical DDC methods while maintaining theoretical convergence guarantees under standard over-parameterization assumptions.

The paper concludes that this ERM approach offers a principled, scalable, and statistically consistent route for offline IRL/DDC, combining the identification rigor of structural econometrics with the computational efficiency of modern gradient-based learning.