🤖 machine learning

Insulin4RL: Real-Time Insulin Management in the Intensive Care Unit for Offline Reinforcement Learning

This paper introduces Insulin4RL, a large-scale offline reinforcement learning dataset derived from MIMIC-IV that features naturally irregular, real-time insulin management decisions from over 12,000 ICU patients to address the limitations of temporally discretized data in clinical research.

Original authors: Thomas Frost, Steve Harris

Published 2026-06-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Thomas Frost, Steve Harris

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to be a doctor. You want the robot to learn how to adjust insulin for patients in the Intensive Care Unit (ICU) so their blood sugar stays safe. To do this, you need to show the robot thousands of examples of real doctors making decisions in the past.

This paper introduces a new "textbook" for that robot called Insulin4RL. Here is the story of why it was made and what it does, explained simply.

The Problem: The "Time-Lapse" Mistake

For a long time, researchers trying to teach AI medical skills have been using a strange trick. Real life is messy. Doctors check a patient's blood sugar at 9:00 AM, then maybe not again until 11:30 AM, then again at 2:15 PM. The time between checks is random.

However, most old computer models couldn't handle that messiness. So, researchers took the real data and chopped it up into neat, fixed blocks of time—like forcing a jagged mountain range into a flat grid of 4-hour squares. They said, "Okay, we'll pretend the doctor made a decision exactly every 4 hours."

The authors of this paper argue that this is like watching a movie where someone has deleted all the frames between the main scenes. You might think you understand the plot, but you've missed all the subtle movements and timing that actually matter. By forcing real, messy data into neat boxes, the computer models learn a "fictional" version of reality. They might look good on paper, but if you put them in a real hospital where time is unpredictable, they could fail.

The Solution: A "Real-Time" Playground

The authors created Insulin4RL, a massive new dataset that refuses to chop up time.

What is it? It's a collection of over 375,000 decisions made by real doctors for over 12,000 patients in the ICU.
What makes it special? It keeps the "jagged edges." It records exactly when a doctor checked the blood sugar and exactly when they changed the insulin drip. If a doctor waited 47 minutes to check, the data waits 47 minutes. If they acted 3 minutes later, the data reflects that.
The Analogy: If the old datasets were like a stroboscope (flashing lights that make movement look jerky and fake), Insulin4RL is like a high-speed video camera that captures every smooth, natural movement of the doctor's hand.

How They Tested It

To prove this new dataset works, the authors ran some experiments:

The "Copycat" Test: They taught a computer to simply copy what the real doctors did (this is called "Behavioral Cloning"). The computer learned very well, proving the data contains clear, logical signals that a machine can understand.
The "Time Travel" Test: They trained two different AI models on the same data.
- Model A was forced to pretend time was fixed (the old way).
- Model B was allowed to see the real, irregular timing (the new way).
- The Result: The two models learned different strategies. The one that respected the real timing (Model B) performed better. This proves that the "fake" time in old datasets actually changes what the AI learns, potentially making it less safe for real patients.

What's Inside the Box?

The dataset isn't just numbers; it's a structured story of a patient's stay:

The Inputs: It includes 140 different things, like blood test results, weight, age, and what other medicines the patient is getting.
The Actions: It categorizes what the doctor did: "Keep the insulin the same," "Stop the insulin," or "Change the insulin rate."
The Outcome: It tracks if the patient survived and how their blood sugar reacted.

The Bottom Line

The authors aren't saying this AI is ready to treat patients tomorrow. Instead, they are saying: "Stop training your robots on fake, chopped-up time."

They have released this new "real-time" dataset to the public so that other scientists can build better, safer AI. By using this dataset, future AI models will learn to handle the unpredictable, messy reality of a hospital, rather than a clean, fictional world that doesn't exist.

Technical Summary: Insulin4RL

Problem Statement

Offline Reinforcement Learning (ORL) holds promise for optimizing clinical decision-making using historical Electronic Health Record (EHR) data. However, current ORL research in healthcare relies heavily on datasets that have been temporally discretized into fixed, regular time intervals (e.g., aggregating events into 4-hour windows). The authors argue that this discretization creates "fictional representations" of complex clinical scenarios, introducing significant bias and uncertainty regarding window size selection. Consequently, models trained and evaluated on such data may learn maladaptive policies that appear beneficial in retrospective analysis but fail in real-world deployment where clinical interventions occur at naturally irregular intervals. The field lacks a benchmark that preserves the stochastic, continuous-time nature of clinical care, particularly for critical care tasks like insulin titration.

Methodology

To address these limitations, the authors introduce Insulin4RL, a curated dataset derived from the MIMIC-IV v3.1 dataset. The methodology focuses on preserving the natural irregularity of clinical data rather than forcing it into fixed time bins.

Cohort Construction: The dataset comprises 12,209 unique adult ICU patients requiring continuous intravenous insulin infusion. An episode is defined as the continuous period surrounding an infusion (including interruptions), padded with 24 hours of pre- and post-infusion data. The data is split 80/10/10 at the patient level for training, validation, and testing.
State Representation: Inspired by the Medical Event Data Standard (MEDS), patient states are represented as raw sequences of retrospective medical event tuples $(f, t, v)$ , where $f$ is the feature code, $t$ is the elapsed time in minutes, and $v$ is the value. The input includes 140 features covering demographics, labs, medications, and physiology. The sequence is capped at 400 events, with older events (>7 days) excluded.
Action Space & Decision Labels: Unlike standard MDPs that assume "do nothing" actions, Insulin4RL defines decision points strictly at recorded blood glucose measurements. Actions are categorized based on clinician behavior within a specific window (5 minutes prior to 30 minutes post-check):
1. Maintain: Insulin rate change < 0.25 units/hr.
2. Stop: Rate drops from >0.1 to <0.1 units/hr.
3. Change: Rate adjusted by >0.25 units/hr (without stopping).
  Changes exceeding 5.5 units/hr are treated as outliers and removed.
Framework: The authors propose modeling the environment as a finite-horizon partially observable Semi-Markov Decision Process (SMDP) rather than a standard Markov Decision Process (MDP). This accounts for variable sojourn times ( $\eta$ ) between decisions and dynamically adjusts the discount factor ( $\gamma^\eta$ ).
Baseline Experiments: The authors implemented a shared sequence-modelling architecture (feature embedding, 1D-CNN, and LSTM) to train:
- Behavioral Cloning (BC): To mimic clinician behavior.
- Implicit Q-Learning (IQL) and Conservative Q-Learning (CQL): Two state-of-the-art model-free ORL algorithms.
- Evaluation: Models were evaluated using Fitted Q-Evaluation (FQE) under both standard MDP assumptions (fixed $\gamma$ ) and SMDP assumptions (dynamic $\gamma^\eta$ ).

Key Contributions

Insulin4RL Dataset: A large-scale resource containing over 375,000 naturally timed decision points for ICU insulin titration. It is the first dataset explicitly designed to evaluate ORL under realistic, irregular clinical sampling assumptions.
SMDP Framework Validation: The paper demonstrates that treating clinical data as an SMDP (preserving irregular timing) yields different policy outcomes compared to standard MDP discretization.
Standardized Evaluation Protocol: The authors provide a baseline evaluation using Fitted Q-Evaluation (FQE) and behavioral cloning, establishing a benchmark for future research in continuous-time healthcare ORL.
Open Resources: The dataset is hosted on PhysioNet, accompanied by Jupyter notebooks and code to reproduce all experiments, including the specific tensor transitions and metadata.

Results

Cohort Characteristics: The dataset includes 13,652 distinct infusion episodes with a median duration of 28 hours (max 35 days). The cohort has a high prevalence of comorbidities (e.g., hypertension, cardiac arrhythmias) and a 28-day survival rate of 91.4%.
Behavioral Cloning Performance: The BC model achieved high discrimination for active interventions (AUROC 0.89–0.94) and reasonable precision in predicting continuous dose adjustments (Mean Absolute Error of 0.61 units/hr).
Latent Space Coherence: UMAP visualization of the learned state space revealed clear gradients corresponding to blood glucose, insulin rates, and episode duration. Notably, the latent space naturally separated surviving and non-surviving patients, despite survival not being an explicit training objective.
ORL Performance: All ORL models (IQL and CQL) exceeded the behavioral cloning baseline. Crucially, models trained and evaluated under the SMDP framework achieved the best overall returns, outperforming their MDP counterparts. This divergence highlights the sensitivity of ORL algorithms to temporal assumptions.

Significance and Claims

The authors claim that the scientific consensus on healthcare ORL is currently limited by the quality of evaluation practices, specifically the reliance on temporally discretized benchmarks. By introducing Insulin4RL, they aim to shift the focus toward realistically timed clinical data.

The paper argues that:

Temporal Assumptions Matter: Identical model architectures and inputs can learn divergent optimal behaviors depending on whether they are trained under MDP or SMDP assumptions.
Risk of Overestimation: The widespread use of discretized datasets with evenly spaced intervals may lead to a general overestimation of model safety and efficacy in real-world applications where decision intervals are unpredictable.
Future Direction: The field requires more evaluation benchmarks that accurately represent decision timing. Insulin4RL serves as a testbed for developing continuous-time ORL algorithms and evaluating new Off-Policy Evaluation (OPE) methodologies in high-dimensional, healthcare-specific settings.

The authors conclude that adopting continuous-time evaluation frameworks is a necessary step toward more rigorous and realistic ORL model assessment, bringing the field closer to the conditions required for real-world clinical trials.