📊 statistics

Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directions

This paper surveys the statistical challenges and methodological advances in bridging the gap between reinforcement learning research and real-world deployment by framing practical application as a cyclical process of online optimization, offline analysis, and iterative redeployment, while outlining future directions for impactful, use-inspired research.

Original authors: Asim H. Gazi, Yongyi Guo, Daiqi Gao, Ziping Xu, Kelly W. Zhang, Susan A. Murphy

Published 2026-07-14

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Asim H. Gazi, Yongyi Guo, Daiqi Gao, Ziping Xu, Kelly W. Zhang, Susan A. Murphy

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to be the perfect personal coach for a human. You want the robot to learn how to nudge, encourage, or help a person make better choices—like walking more, studying harder, or eating better. This is the world of Reinforcement Learning (RL).

In video games, this is easy. The robot can play the game a billion times, crash into walls, lose lives, and try again until it becomes a god-like champion. It has a "perfect simulator" where it can make mistakes without hurting anyone.

But in the real world, especially when humans are involved, things get messy. This paper argues that we can't just copy-paste the video game strategy. Instead, we need a new playbook that treats learning as a three-part loop: learning while acting, pausing to analyze, and starting over again with new ideas.

Here is the story of why the old way fails and what the new way looks like.

The Two Big Roadblocks

The authors point out two massive hurdles that stop RL from working well in real life:

The "No Sandbox" Problem: In games, you can crash a car a million times to learn how to drive. In real life, if you are coaching a person, you can't let them crash. You can't bombard them with thousands of bad suggestions just to see what happens. You have very few chances to interact with them before they get annoyed and quit. The paper calls this a lack of "extensive interaction."
The "Moving Target" Problem: The world changes. A person's life, their health, or the technology they use evolves. A strategy that worked last year might be useless today. The paper notes that environments often undergo "substantial changes," meaning you can't just train a robot once and leave it alone; you have to keep redesigning it.

The Three-Part Loop: The "Coach's Cycle"

Instead of a straight line from "Training" to "Winning," the paper suggests we view real-world RL as a continuous cycle with three distinct phases:

The "On-the-Job" Phase (Online Learning): The robot is out in the field, making decisions in real-time. It's trying to help the person right now. But because it can't make too many mistakes, it has to be careful. It's like a student driver who has to drive carefully because they only have one tank of gas.
The "Debrief" Phase (Offline Analysis): After a while, the robot stops. It takes all the data it collected and sits down with a team of human experts to study it. They ask: "What worked? What didn't? Why?" This is where they use statistics to figure out the truth without risking anyone's safety.
The "Redesign" Phase (Iterative Deployment): Based on the debrief, the team changes the robot's brain. Maybe they add new rules, remove bad ideas, or tweak how it learns. Then, they send the new robot out for another round.

The paper emphasizes that this isn't just "learning while acting." It's a cycle where you learn, stop, analyze, change, and start again.

The Balancing Act: Bias vs. Variance

One of the trickiest parts the paper discusses is a concept called the Bias-Variance Tradeoff. Let's use a metaphor: The Crystal Ball vs. The Guessing Game.

The Crystal Ball (Low Bias, High Variance): Imagine you try to build a super-accurate crystal ball that predicts exactly what a specific person will do. To do this, you need a huge amount of data. But in the real world, you only have a tiny bit of data. If you try to build this complex crystal ball with so little data, it will be "noisy." It might predict the person will walk 10,000 steps today, then 50 steps tomorrow, then 10,000 again, just because it's confused. It's too sensitive to the small amount of data it has.
The Guessing Game (High Bias, Low Variance): To fix the noise, you might decide to use a simpler rule: "Everyone walks 5,000 steps." This is a "biased" guess because it's not perfectly true for everyone. But it's stable. It won't swing wildly.

The paper suggests that in real-world settings with limited data, we often have to accept a little bit of "bias" (using simpler rules or borrowing ideas from other people) to keep the system stable and safe. We can't afford the wild swings of a complex model that hasn't seen enough data yet.

The "Digital Twin" Idea

The paper also talks about a cool idea called Digital Twins. Imagine you have a video game version of the real person you are coaching. You can't experiment on the real person, but you can experiment on the "Digital Twin."

The paper suggests we can build these twins using data from past deployments. We can test new ideas on the twin to see if they work before we try them on the real human. However, the authors are careful to note that this is still an emerging idea. They mention that while some researchers are building these twins (like the "JITAI-Twin" framework), we need more work to make sure these simulations are accurate enough to trust.

What the Paper Says We Can't Do (Yet)

It's important to know what this paper says is not the solution:

It's not just "more data": You can't just wait for more data to solve the problem because, in many real-world cases (like health), you simply cannot get more data without harming or annoying the person.
It's not "perfect models": The paper argues against the idea that we can always build a perfect, complex model of how humans work. Sometimes, simpler, slightly "wrong" models are better because they are more stable.
It's not a one-time fix: You cannot train an AI once and deploy it forever. The paper explicitly rules out the idea of a "set it and forget it" system for human interactions because the environment changes too much.

The HeartSteps Story: A Real-World Example

To prove this isn't just theory, the authors look at a real project called HeartSteps. This was an app designed to get people to walk more.

Version 1: They didn't use a smart robot yet. They just randomly told people to walk or not to walk to gather data.
Version 2: They used that data to build a smarter robot. But they didn't make it too complex. They used a "simplified" model that borrowed ideas from the first version to avoid making wild guesses.
The Result: The robot got better over time, but only because they kept stopping to analyze the data, redesign the robot, and try again.

The Bottom Line

The paper suggests that for Reinforcement Learning to truly help humans, we need to stop treating it like a video game where you just play until you win. Instead, we need to treat it like a scientific experiment that never really ends.

We need to be humble about our data, willing to use simpler models to stay safe, and ready to constantly redesign our systems as the world changes. The authors suggest that the future of RL isn't about finding the single "perfect" algorithm, but about building a cycle of learning, analyzing, and improving that respects the limits of real life.

They don't claim this is a solved problem. In fact, they highlight that we are just starting to figure out how to do this safely and effectively. But by following this three-part loop, we might finally bridge the gap between cool computer science and real-world help.

Technical Summary: Reinforcement Learning in the Real World

Problem Statement

Reinforcement Learning (RL) has achieved state-of-the-art performance in domains with extensive interaction capabilities and stable environments (e.g., gaming, robotics via high-fidelity simulators). However, a substantial gap exists between these successes and the deployment of RL in real-world settings involving human interaction, such as healthcare, education, and public health. The paper identifies two recurring challenges that hinder this transition:

Challenge C1 (Data Scarcity): Many real-world settings do not allow for extensive interaction with the target environment due to practical constraints (e.g., risk of harm, user burden, disengagement). This limits the ability of agents to learn solely through online exploration, particularly when adapting to new individuals or populations.
Challenge C2 (Non-stationarity and Redesign): Target environments often undergo substantial changes over time (e.g., shifts in population characteristics, technological advancements, or evolving societal norms). These changes necessitate the redesign and redeployment of RL systems, rather than relying on a single static policy learned offline.

The paper argues that bridging this gap requires a shift from purely theoretical regret minimization to a framework that integrates online learning, offline statistical inference, and iterative system improvement.

Proposed Framework: A Three-Component Process

The authors frame the application of RL in practice as a continuous, three-component process (illustrated in Figure 1 of the paper):

Within-Deployment Online Learning and Optimization: The phase where an RL algorithm interacts with the environment, collects data, and updates its policy in real-time.
Between-Deployment Offline Learning and Inference: The phase where data collected from prior (or ongoing) deployments is analyzed to estimate causal effects, evaluate policies, and inform the design of subsequent deployments.
Iterative Deployment–Redeployment: A feedback loop where insights from offline analysis and human oversight guide the redesign and redeployment of the RL algorithm for continual improvement.

The paper emphasizes that while these phases are depicted sequentially for clarity, in practice, online and offline phases often co-occur (e.g., using data from time $t_0$ for offline learning while the algorithm continues to deploy for $t > t_0$ ).

Methodology and Key Technical Areas

The paper provides a narrative review of statistical and methodological developments addressing the challenges within the three-component framework.

1. Within-Deployment: Managing Bias-Variance Tradeoffs (Addressing C1)

Given data scarcity, the paper highlights the necessity of managing the bias-variance tradeoff to ensure sample efficiency and stability.

Pooling and Regularization: Strategies include pooling data across heterogeneous individuals (trading individual bias for reduced variance) and simplifying model structures (e.g., using shorter learning horizons or smaller discount factors).
Causal RL: The integration of causal knowledge (via Causal DAGs) is proposed as a mechanism to navigate bias-variance tradeoffs. Causal structures can:
- Construct informative surrogate rewards for distal outcomes.
- Identify relevant state variables and action components to reduce dimensionality.
- Embed prior knowledge into Bayesian priors or counterfactual data augmentation.
Model Misspecification: The authors note that intentionally using simpler, potentially misspecified models can stabilize exploration and ensure policy convergence, which is critical for downstream statistical inference.
Large Language Models (LLMs): LLMs are discussed as tools to mitigate data scarcity by inferring latent states from unstructured text (e.g., patient self-reports) or acting as in-context decision-making agents, though their deployment in live online loops for long-term optimization remains nascent.

2. Between-Deployment: Statistical Inference and Offline Learning (Addressing C2)

The paper reviews methods for analyzing adaptively collected data to support the redesign of RL systems.

Inference with Pooling: Standard statistical inference methods often fail when RL algorithms pool data across individuals because the pooling mechanism induces dependencies between user trajectories. Recent work (e.g., Zhang et al.) provides conditions under which asymptotic normality holds for specific smoothing policies, though robust inference for general pooling algorithms remains an open challenge.
Offline Algorithm Selection and Initialization: Offline data from prior deployments can be used to:
- Select Algorithms: Using simulators (e.g., k-nearest neighbor, rejection sampling) or digital twin frameworks (e.g., JITAI-Twin) to evaluate candidate algorithms before deployment.
- Warm-Start: Pre-training policies or value functions on offline data to improve early-stage online performance (Hybrid Offline-Online RL, Meta-RL).
Tradeoffs: The paper highlights the tension between minimizing cumulative regret during a deployment (optimization) and maximizing the quality of data for subsequent offline policy learning or causal inference (exploration).

3. Iterative Deployment and Continual Improvement

The paper discusses the need for algorithms that adapt to distribution shifts and new tasks over time.

Non-stationarity: Algorithms must adapt to changing environment dynamics, often requiring a balance between retaining past knowledge (stability) and learning new tasks (plasticity) to avoid catastrophic forgetting.
Continual Learning: This emerging area focuses on designing algorithms that perform well across a sequence of changing tasks, requiring new benchmarks and simulation environments that reflect real-world dynamics.

Key Contributions

Conceptual Framework: The paper proposes a unified, three-component framework (Online, Offline, Iterative) that moves beyond the traditional separation of online and offline RL, emphasizing the feedback loop required for real-world deployment.
Use-Inspired Review: Rather than a systematic survey of all RL subfields, the paper curates methodological developments specifically relevant to human-interaction settings, focusing on statistical challenges like sample efficiency, valid inference, and handling non-stationarity.
Integration of Causal Inference: It explicitly connects causal inference methods (e.g., causal DAGs, mediators) with RL design to address data scarcity and improve sample efficiency.
Case Study Application: The paper applies its framework to the HeartSteps JITAI (Just-in-Time Adaptive Intervention), detailing how statistical compromises (bias-variance tradeoffs), causal reasoning, and iterative redesign were utilized across four distinct deployments targeting different populations.

Results and Claims

The paper does not present new experimental results or a single unified algorithm. Instead, its "results" are the synthesis of existing literature and the identification of gaps:

Current Limitations: There is a lack of real-world deployments that explicitly leverage causal DAGs for online learning efficiency. Furthermore, there are no standardized benchmarks for evaluating offline algorithm selection or offline-to-online learning under domain shifts common in healthcare.
Open Problems: The authors identify several critical open questions, including:
- How to construct algorithms that optimally trade off bias and variance using causal inductive biases.
- How to develop robust statistical inference methods for data generated by pooling RL algorithms.
- How to design continual learning algorithms that degrade gracefully under misspecified assumptions.
- The need for "digital twin" frameworks that evolve alongside real-world deployments.

Significance

The paper claims that for RL to make a substantial impact in high-stakes real-world applications (particularly those involving humans), the community must move beyond minimizing regret in static environments. The significance lies in:

Bridging the Gap: Providing a roadmap for integrating statistical methodology, domain knowledge, and algorithm design to address the specific constraints of human-centric RL.
Scientific Rigor: Emphasizing the need for valid statistical inference and replicability in adaptive experiments, ensuring that data collected during deployment can be used to generate generalizable scientific knowledge.
Continual Improvement: Framing RL deployment not as a one-time optimization task but as a statistical learning process that unfolds over multiple cycles of deployment, analysis, and human-guided redesign.

The authors conclude that realizing the potential of RL in these settings requires a research agenda that prioritizes continual learning, replicability, and scientific interpretability alongside algorithmic performance.