📊 statistics

Learning from the Unseen: Offline Reinforcement Learning with Hidden Actions

This paper introduces LURE, the first method for offline reinforcement learning with hidden actions, which leverages next-state variables as proxies to enable multiply robust and statistically valid policy value estimation in infinite-horizon Markov decision processes.

Original authors: Zeyu Bian, Ying Zhou, Yifan Cui

Published 2026-07-29

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Zeyu Bian, Ying Zhou, Yifan Cui

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to play a complex game, like navigating a city or managing a patient's health. You don't have the robot run through the game in real-time; instead, you give it a giant logbook of past attempts made by a human player. This is the world of Reinforcement Learning (RL), a branch of artificial intelligence where computers learn by trial and error to make the best possible decisions over time. The goal is usually to figure out: "If we followed a specific set of rules (a policy) in the future, how well would we do, based on this old logbook?"

However, there's a catch. In the real world, logbooks are rarely perfect. Sometimes, the person writing the notes made a mistake, or the system recording the data was glitchy. In the language of this paper, the "action" the human actually took (like giving a specific medicine) might be hidden, and all we see is a "proxy" or a noisy guess of what happened (like a doctor's note that says "gave medicine" when they actually didn't, or vice versa). If you try to teach your robot using a logbook full of these errors, it will learn the wrong lessons, leading to bad decisions later. This paper tackles the tricky problem of how to learn effectively when the most important part of the story—the true action—is missing or disguised.

The Mystery of the Missing Move

In the paper titled "Learning from the Unseen," the authors, Zeyu Bian, Ying Zhou, and Yifan Cui, face a detective story where the main clue is missing. Imagine you are a detective trying to solve a crime by looking at a security video. But here's the twist: the video shows the suspect's shadow, not the suspect themselves. You can see the shadow move, but you don't know exactly what the suspect's hands were doing. In the world of data science, this is called having "hidden actions."

Usually, when scientists try to evaluate a strategy using past data (a process called Off-Policy Evaluation), they assume the logbook is perfect. They assume that if the log says "Action A was taken," then Action A was definitely taken. But in messy real-world scenarios—like hospital records where a doctor might write down a treatment hours later, or a system that accidentally mislabels a button press—this assumption is a trap. If you ignore the errors, your evaluation of the strategy will be biased, like judging a chef's cooking skills based on a menu that has the wrong ingredients listed.

The Magic of the Next Step

The authors' big breakthrough is realizing that even if you can't see the "action" directly, you can often see its shadow in the next moment.

Think of it like this: If a magician waves a wand (the hidden action), you might not see the wand clearly because of a smudge on the camera. But you can see the rabbit appear in the hat (the next state). The rabbit didn't just appear out of nowhere; it appeared because of the wand wave. By studying the relationship between the "next state" (the rabbit) and the "noisy note" (the smudged video), the authors figured out how to mathematically reconstruct what the magician actually did.

They call their new method LURE (Learning from the Unseen: Robust Estimator). It's like a super-smart detective who doesn't just look at the blurry photo of the suspect but also looks at the footprints left behind and the weather report to deduce exactly what happened.

How LURE Works: The "Double-Check" System and the Detective's Toolkit

The paper introduces a clever way to estimate the value of a strategy without getting fooled by the errors. They use a concept called Multiple Robustness.

Imagine you are trying to guess the temperature outside, but your thermometer is broken. You have three other clues: a wet dog, a swaying tree, and a cloud formation.

If your thermometer is broken, you can still guess correctly if your "wet dog" clue is accurate.
If the dog is dry but the tree is swaying, you can still guess correctly.
The system is "robust" because it doesn't need every clue to be perfect; it just needs some combination of them to be right.

LURE works the same way. It builds a mathematical model that checks several different parts of the data. Even if the model for the "noisy action" is slightly wrong, or the model for the "next state" is slightly off, the estimator can still find the true value of the strategy as long as at least one of the other parts is correct. This makes the final answer much more reliable than previous methods, which would crumble if just one part of the data was messy.

But there's a hidden complexity in how LURE solves this puzzle. Since the true actions are never seen, the computer has to guess the probabilities of what actually happened. To do this, the authors developed a special iterative algorithm (based on a method called Expectation-Maximization, or EM). Think of this as a detective who keeps refining their theory:

The Guess (E-step): The computer starts with a guess about what the hidden actions were.
The Update (M-step): Using that guess, it updates its models for how actions lead to rewards and next states.
The Refinement: It then uses these updated models to make a better guess about the hidden actions, and repeats the cycle until the story makes perfect sense.

However, there's one final twist. Because the computer is guessing, it might accidentally swap the labels. It might decide that "Action 0" is actually the "Medicine" and "Action 1" is "No Medicine," when in reality, it's the exact opposite. The math works either way, but the meaning is flipped. To fix this, the authors include a crucial label alignment step. Before giving the final answer, the system checks which "guess" makes the most sense with the noisy data (for example, which hidden action is more likely to result in the observed "Medicine" note). It then flips the labels if necessary to ensure the final report matches reality.

Testing the Theory

The authors didn't just write down a theory; they put it to the test in three different ways:

Simple Simulations: They created a tiny, made-up world (a "Tabular MDP") with just three states and two actions. They intentionally messed up the data by mislabeling the actions 5% to 30% of the time. While other methods got confused and gave wrong answers, LURE stayed on target, correctly guessing the value of the strategy.
Complex Simulations: They moved to a more complex, continuous world (like a robot moving in a smooth space) and found the same result: LURE handled the errors gracefully, while others failed.
Real-World Data: They tested LURE on a massive database of real patient records from a hospital (MIMIC-III), specifically looking at how to treat sepsis (a life-threatening reaction to infection). In this real-world scenario, they compared LURE against standard methods. The results were striking: the standard methods suggested that one treatment was better than another, but LURE, by accounting for the hidden errors in the medical records, suggested a different, more reliable ranking. In fact, the confidence intervals (the range of likely answers) from the standard methods overlapped so much they couldn't tell the difference, while LURE provided a clear, distinct answer.

The Takeaway

The paper concludes that ignoring hidden actions is a dangerous gamble. By using the "next state" as a natural clue to uncover the truth, and by building a system that is robust to mistakes in different parts of the model, we can finally evaluate strategies accurately even when our data is imperfect.

This isn't just a theoretical win; it's a practical tool for the future. Whether it's deciding on medical treatments, managing traffic lights, or training AI agents, LURE offers a way to learn from the past without being misled by the noise. The authors show that with the right mathematical detective work, we can see the unseen and make better decisions for tomorrow.

Technical Summary: Learning from the Unseen: Offline Reinforcement Learning with Hidden Actions

1. Problem Formulation

Standard offline Reinforcement Learning (RL) and Off-Policy Evaluation (OPE) algorithms typically assume that the actions recorded in the dataset are observed without error. However, in many real-world applications—particularly in healthcare using Electronic Health Records (EHR)—the true actions ( $A_t$ ) are unobserved (latent). Instead, only noisy proxies or surrogates ( $\tilde{A}_t$ ) are available due to annotation errors, delayed charting, imperfect management systems, or privacy-preserving data coarsening.

The paper addresses the problem of evaluating a target policy $\pi$ in an infinite-horizon discounted Markov Decision Process (MDP) where the action space is partially or fully latent. The goal is to estimate the policy value $V(\pi)$ using historical data generated by a behavior policy $b$ , where the observed data consists of trajectories $\{(S_t, \tilde{A}_t, R_t, S_{t+1})\}$ rather than the true $\{(S_t, A_t, R_t, S_{t+1})\}$ . Naively substituting the observed surrogate $\tilde{A}_t$ for the true action $A_t$ in existing OPE methods leads to biased estimates because the Q-function and density ratios are defined with respect to the latent true action.

2. Methodology

2.1 Identification Strategy

The authors establish that the policy value $V(\pi)$ is identifiable from the observed data distribution under specific assumptions. The core insight is leveraging the next state ( $S'$ ) as a natural proxy for the unobserved action. Since the transition dynamics depend on the true action, $S'$ contains information about the hidden action that can be exploited for identification.

The identification relies on four key assumptions (Assumption 3.1):

Coverage: The support of the target policy is contained within the support of the behavior policy.
Relevance: The reward $R$ , the surrogate $\tilde{A}$ , and the next state $S'$ are not conditionally independent of the true action $A$ given the state $S$ .
Conditional Independence: Given the true action $A$ and state $S$ , the surrogate $\tilde{A}$ , reward $R$ , and next state $S'$ are mutually independent.
Surrogate Quality: The surrogate provides more information than a random guess (e.g., for binary actions, the probability of observing $\tilde{A}=0$ is lower when $A=1$ than when $A=0$ ).

Under these conditions, the authors prove (Theorem 1) that $V(\pi)$ is uniquely identified.

2.2 The LURE Estimator

To estimate the value, the paper proposes LURE (Learning from the Unseen: Robust Estimator). The estimator is constructed based on an observed-data influence function derived from the full-data efficient influence function.

The influence function $\phi_\pi(O)$ is defined as:
$\phi_\pi(O) = E_{p_e}(V^\pi(S)) + (1-\gamma)^{-1} \sum_{a} g'_a(S', \tilde{A}, S)\omega(S, a)[R - \theta_R(S, a)] + \frac{\gamma}{1-\gamma} \sum_{a} g_a(R, \tilde{A}, S)\omega(S, a)[V^\pi(S') - M V^\pi(S, a)] - V(\pi)$
where:

$\omega(S, a)$ is the marginal density ratio between the target and behavior policies.
$\theta_R(S, a)$ is the reward function.
$M V^\pi(S, a)$ is the expected future value.
$g_a$ and $g'_a$ are constructed functions acting as "bridge" estimators that replace the indicator function $\mathbb{I}(A=a)$ using the observed surrogate $\tilde{A}$ and the next-state proxy $S'$ . Specifically, they are ratios involving the conditional expectations of $\tilde{A}$ and $S'$ given $S$ and $A$ .

2.3 Estimation Procedure (EM Algorithm)

Since the true action $A$ is latent, standard supervised learning cannot be applied directly to estimate the nuisance functions ( $\omega, \theta_R, \theta_{\tilde{A}}, \theta_{S'}, Q^\pi$ ). The authors propose a novel Expectation-Maximization (EM) style iterative algorithm (Algorithm 2):

Initialization: Initialize the posterior probability $\eta(O; a) = P(A=a | O)$ .
M-Step (Weighted Supervised Learning): Treat the posterior $\eta$ as known weights. Update the nuisance functions (behavior policy, measurement model, reward/transition models, and density ratio) using weighted regression or weighted minimax optimization (for the density ratio $\omega$ ).
E-Step (Posterior Update): Update the posterior probability $\eta(O; a)$ using Bayes' rule and the updated nuisance estimates.
Iteration: Repeat until convergence.

Additionally, the paper addresses label alignment, as the EM algorithm may arbitrarily swap the labels of the latent classes (e.g., mapping true $A=0$ to component 1). A label-alignment step is performed post-estimation based on the conditional probability of the surrogate to ensure consistent interpretation.

2.4 Cross-Fitting

To mitigate overfitting bias and control empirical-process remainders, the authors employ a $K$ -fold cross-fitting procedure. Nuisance functions are estimated on training folds and evaluated on held-out folds.

3. Key Theoretical Contributions

Multiple Robustness: The LURE estimator is multiply robust. As shown in Corollary 1, the estimator remains consistent ( $\hat{V}(\pi) - V(\pi) = o_p(1)$ ) if at least one of several specific combinations of nuisance components is correctly specified. For example, consistency holds if the reward bridge $\theta_R$ is consistent along with either the latent-action regression $\theta_{\tilde{A}}$ or the next-state regression $\theta_{S'}$ , provided the continuation value error is negligible.
Asymptotic Normality: Under regularity conditions and convergence rates of the nuisance estimators (specifically requiring the product of errors to decay faster than $n^{-1/2}$ ), the estimator is asymptotically normal (Theorem 4). This enables the construction of valid Wald-type confidence intervals.
Error Bounds: The paper provides finite-sample error bounds (Theorem 3) that decompose the estimation error into the influence function average, a label-selection remainder, and a second-order remainder involving products of nuisance estimation errors.

4. Experimental Results

The authors evaluate LURE through simulations and a real-world application:

Tabular and General MDPs: In synthetic environments with varying misclassification rates ( $\tau \in \{0.05, 0.1, 0.2, 0.3\}$ ), standard OPE methods (SIS, FQE, LSTD, MIS, DRL) that ignore the latent structure exhibit significant bias that worsens with higher misclassification. LURE remains centered around the true policy value.
OpenAI Gym (CartPole): In a continuous-state environment, LURE outperforms baselines, maintaining accurate coverage rates for confidence intervals (close to the nominal 95% level), whereas baselines show substantial bias or unstable coverage.
MIMIC-III Sepsis Analysis: The method is applied to a sepsis management dataset. The study evaluates three policies: "Always IV," "No IV," and a "SOFA-tailored" rule.
- LURE estimates the "Always IV" policy as having the lowest expected SOFA score (62.10), suggesting it is the optimal policy.
- The 95% confidence interval for "Always IV" [60.16, 64.04] does not overlap with the intervals for the other policies.
- In contrast, the standard DRL estimator produces overlapping confidence intervals for all policies, failing to distinguish between them. This highlights that ignoring hidden actions can lead to misleading empirical conclusions in clinical settings.

5. Significance and Claims

The paper claims to be the first work to study offline reinforcement learning in the hidden-action setting. Its significance lies in:

Addressing a Critical Gap: It provides a principled framework for OPE when action data is noisy or latent, a common issue in EHR-based studies that standard RL methods cannot handle reliably.
Theoretical Rigor: It establishes identification conditions and derives an influence function that accounts for the sequential nature of RL, distinguishing it from static hidden-treatment frameworks.
Practical Utility: By demonstrating multiply robustness and asymptotic normality, LURE enables valid statistical inference (confidence intervals and hypothesis testing) in settings where previous methods would yield biased point estimates.
Real-World Impact: The application to the MIMIC-III database illustrates that accounting for hidden actions can fundamentally alter policy rankings and clinical recommendations, suggesting that standard RL evaluations in healthcare may be unreliable without such corrections.

The authors maintain a modest tone, noting that while the framework is generalizable to categorical actions, the current work focuses on binary actions for clarity, and the method relies on the specific assumption that the next state serves as a valid proxy for the hidden action.