The hidden risks of temporal resampling in clinical reinforcement learning

1. Problem Statement

Reinforcement Learning (RL) promises the optimization of clinical decision-making, yet direct online interaction with patients is unsafe. Consequently, researchers rely on Offline Reinforcement Learning (ORL), where models are trained on retrospective data from electronic health records (EHR).

A widespread preprocessing step in current healthcare ORL pipelines is temporal resampling (binning). Since clinical records are inherently irregular (e.g., vital signs every 15 minutes, insulin doses every 2 hours), researchers frequently aggregate or interpolate data into fixed, uniform time intervals (e.g., 1-hour or 4-hour bins) to fit them into standard Markov Decision Process (MDP) frameworks.

The Core Hypothesis: The authors argue that this discretization violates a fundamental assumption of ORL: that training data must faithfully reflect the deployment environment. By forcing irregular clinical events into artificial, uniform intervals, researchers create a "fictional" representation of reality. This paper posits that:

1. Temporal resampling introduces counterfactual trajectories (reversing the causal order of events).
2. Models trained on resampled data perform suboptimally in real (irregular) environments.
3. Standard retrospective evaluation methods on resampled data systematically overestimate model performance, creating a false sense of safety and efficacy.

2. Methodology

To rigorously test these hypotheses without endangering patient safety, the authors conducted an in-silico clinical study using a controlled, FDA-approved simulator.

Experimental Setup

• Environment: The UVA/Padova Type 1 Diabetes Simulator (30 virtual patients) and a Gridworld navigation task (LavaGap).
• Data Generation:
  • An "expert" agent (trained via Proximal Policy Optimization, PPO) interacted with an irregular version of the environment, where decision intervals were stochastic (variable time steps).
  • This generated a "Ground Truth" dataset with natural, irregular decision timestamps.
• Dataset Variants: The Ground Truth dataset was processed into three versions:
  1. Raw: Original irregular intervals were preserved (modeled as a Semi-Markov Decision Process, SMDP).
  2. Interpolated: Upsampled to a fixed high frequency (e.g., every 10 minutes), treating every step as a decision.
  3. Binned: Downsampled/aggregated into fixed windows (2-hour and 4-hour intervals), averaging observations and summing rewards.
• Algorithms: Three state-of-the-art Offline RL algorithms were trained on each dataset variant:
  • Behavioral Cloning (BC): Supervised learning baseline.
  • Implicit Q-Learning (IQL): Model-free, off-policy.
  • Conservative Q-Learning (CQL): Model-free, off-policy with distributional constraints.
• Evaluation:
  • Prospective (Online): Agents were deployed back into the irregular simulator to measure true performance.
  • Retrospective (Off-Policy): Fitted Q-Evaluation (FQE) was used to estimate performance on static datasets to determine if standard evaluation methods detect the degradation.

3. Key Contributions and Results

A. Creation of Counterfactual Trajectories

Temporal binning does not merely smooth data; it fundamentally alters causal relationships.

• Mechanism: When observations are averaged over a window preceding an action, the model may see an action (e.g., insulin increase) before the event that necessitated it (e.g., carbohydrate intake) if the events lie on opposite sides of the bin boundary.
• Result: The model learns incorrect cause-and-effect dynamics (e.g., "insulin causes low blood sugar" instead of "low blood sugar triggers insulin release"), leading to flawed policies.

B. Severe Performance Degradation

Models trained on resampled data failed significantly when deployed in the natural, irregular environment:

• Performance Drop: Models trained on 4-hour binned data performed up to 60% worse than those trained on raw data.
• Baseline Failure: All agents trained on 4-hour binned data performed worse than the dataset baseline (the expert policy that generated the data).
• Interpolation Issues: Even "interpolated" data (upsampled to high frequency) performed worse than raw data, likely due to increased sensitivity to distributional shifts and accumulating errors when actions are treated as independent rather than persistent.

C. The "Overly Optimistic" Evaluation Trap

The most critical finding is the failure of retrospective evaluation:

• FQE Bias: When FQE was used to evaluate agents trained on binned data, it predicted performance 1.5 to 3 times higher than the agents actually achieved in the live environment.
• Reason: The evaluation metric was computed on the same distorted (binned) data. The model appeared optimal within the fictional world of binned data, but this optimality did not transfer to reality.
• Implication: A model could pass all retrospective safety checks and be approved for a clinical trial, only to fail catastrophically in practice because the evaluation was blind to temporal misalignment.

4. Significance and Recommendations

Scientific Impact

This study provides empirical evidence that the widespread practice of temporal binning in healthcare ORL is not merely a convenience but a safety risk. It challenges the validity of hundreds of existing studies (the authors reviewed 50 papers, 43 of which used resampling) that rely on retrospective evaluation of binned data.

Clinical Implications

• Patient Safety Risk: Deploying models trained on binned data could lead to suboptimal or harmful treatments (e.g., incorrect insulin dosing), as the model cannot cope with the unpredictability of real clinical timelines.
• Justification of Trials: Falsely optimistic retrospective evaluations could lead to the premature initiation of human clinical trials for models that are fundamentally flawed.

Recommendations for Future Research

1. Abandon Standard Binning: Researchers should stop forcing irregular clinical data into uniform time steps.
2. Adopt SMDP Frameworks: Utilize Semi-Markov Decision Process (SMDP) frameworks that explicitly model variable time intervals between decisions.
3. Preserve Natural Timestamps: Training datasets should retain the natural intervals between decision labels to ensure the model learns the true temporal dynamics of the clinical environment.
4. Robust Evaluation: Retrospective evaluation must be conducted on data matching the intended deployment environment (irregular intervals), not on preprocessed training data.

Conclusion

The paper concludes that temporal resampling creates a fundamental misalignment between training and deployment. By prioritizing the preservation of natural decision intervals and utilizing SMDP frameworks, the field can move toward safer, more effective clinical AI models that are robust to the inherent irregularity of human healthcare.