🤖 AI

Recurrent Deep Reinforcement Learning for Chemotherapy Control under Partial Observability

This paper demonstrates that recurrent deep reinforcement learning policies, which incorporate memory mechanisms like LSTMs, significantly outperform standard feed-forward approaches in optimizing chemotherapy dosing under partial observability by achieving more stable tumor suppression and better normal-cell preservation.

Original authors: Firas Mohamed Elamine Kiram, Imane Youkana, Rachida Saouli, Gian Antonio Susto, Laid Kahloul

Published 2026-05-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Firas Mohamed Elamine Kiram, Imane Youkana, Rachida Saouli, Gian Antonio Susto, Laid Kahloul

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to cook a very delicate dish (the patient's body) that needs to be perfectly seasoned to kill a specific bad ingredient (the tumor) without ruining the good ingredients (healthy cells). The problem is, you can't see inside the pot. You only get to peek at a few clues on the surface, and sometimes those clues are blurry or covered in static.

This paper is about teaching a computer chef how to make the best decisions in this "blind" cooking scenario using a special kind of memory.

The Problem: Cooking in the Dark

In real-world chemotherapy, doctors often have to guess how a patient is reacting because they can't see every single cell in the body. They have to decide how much medicine to give based on limited, noisy information.

Most computer programs that try to solve this (called "Reinforcement Learning") usually assume the chef can see everything inside the pot. But in reality, the "normal cell count" is hidden. The computer only sees the tumor size, immune cells, and drug levels, and even those numbers are a bit fuzzy.

The Solution: Giving the AI a Memory

The researchers tested two types of AI chefs:

The "Forgetful" Chef (Standard AI): This chef looks only at the current snapshot of the pot. If the soup looks okay right now, it makes a decision. It doesn't remember what happened five minutes ago.
The "Remembering" Chef (Recurrent AI): This chef has a memory. It looks at the current snapshot plus a video of what happened in the last few minutes. It remembers, "Oh, the soup looked okay, but I added too much salt five steps ago, so I need to be careful now."

The researchers used a specific type of memory called an LSTM (Long Short-Term Memory), which acts like a mental notebook that helps the AI track how things change over time.

The Experiment: The Blind Taste Test

The team put both chefs to work in a simulated kitchen (a computer model of cancer treatment) with two different rules:

Rule A (Full Visibility): The chef can see everything inside the pot.
Rule B (Partial Visibility): The chef is blindfolded and can only see blurry, noisy hints.

What happened?

In the Full Visibility kitchen: Both chefs did a decent job. The "Forgetful" chef could do almost as well as the "Remembering" one because it had all the information it needed right in front of its eyes.
In the Blindfolded kitchen: The results changed dramatically.
- The Forgetful Chef got confused. It made wild guesses, sometimes giving too much medicine and hurting the healthy cells, or too little and letting the tumor grow. Its performance was all over the place—sometimes great, sometimes terrible.
- The Remembering Chef stayed calm. By looking at the history of what happened, it could figure out what was really going on inside the pot, even if the current clues were blurry. It gave more consistent doses, killed the tumor more reliably, and protected the healthy cells much better.

The Key Takeaway

The paper found that when information is incomplete or noisy (like in real medical situations), having a memory is a game-changer.

The "Remembering" AI didn't just do slightly better; it was much more stable. While the "Forgetful" AI's performance swung wildly like a pendulum, the "Remembering" AI stayed steady. It learned that to make a good decision today, you often need to remember what you did yesterday.

Why This Matters (According to the Paper)

The authors emphasize that this approach is particularly useful because it can learn from past records (like old patient files) rather than needing to experiment on live patients. This is like a chef learning from a logbook of past recipes rather than tasting every new dish to see if it works.

Important Note: The paper explicitly states that this was all done in a computer simulation. They did not test this on real people yet. They also kept the "recipe" (how drugs move in the body) the same for everyone to isolate the effect of memory, meaning real-world patients with different body types weren't part of this specific test.

In short: When you can't see the whole picture, an AI that remembers the past makes much safer and more effective decisions than one that only looks at the present.

Technical Summary: Recurrent Deep Reinforcement Learning for Chemotherapy Control under Partial Observability

Problem Formulation
Chemotherapy dose optimization is framed as a Dynamic Treatment Regime (DTR) problem, requiring sequential decisions that balance tumor suppression against toxicity to healthy cells. While Reinforcement Learning (RL) offers a promising framework for such non-linear, uncertain environments, most existing approaches assume full observability of the patient's physiological state. In clinical reality, however, the full internal state is rarely observable; decisions must be made based on partial, noisy clinical data. This paper investigates the efficacy of memory-augmented policies in addressing this Partial Observability Markov Decision Process (POMDP) setting, specifically isolating the effects of hidden states and observation noise from inter-patient variability.

Methodology
The study utilizes the AhnChemoEnv benchmark from DTR-Bench, which models cancer dynamics via a four-state non-linear Ordinary Differential Equation (ODE) system representing normal cells ( $N$ ), tumor cells ( $T$ ), immune cells ( $I$ ), and drug concentration ( $B$ ).

POMDP Setup: To simulate partial observability, the normal cell population ( $N$ ) is hidden from the agent. Observable variables ( $T, I, B$ ) are corrupted by multiplicative uniform noise, and the ODE derivatives are perturbed by Gaussian noise. Pharmacokinetic/Pharmacodynamic (PK/PD) parameters are held fixed across experiments to ensure that performance differences arise solely from the handling of partial observability and noise, rather than inter-patient heterogeneity.
Architecture: The authors propose a Recurrent TD3 (rTD3) agent. This architecture employs separate Long Short-Term Memory (LSTM) encoders for the actor and critic networks.
- The Actor processes the interaction history ( $\tau_t$ ) via an LSTM to generate a latent representation, which is combined with a direct embedding of the current observation to output the drug dose. This "shortcut" connection preserves immediate information while the LSTM infers hidden dynamics.
- The Critic similarly uses an LSTM to estimate action-values based on history and the current state-action pair.
- Training utilizes a replay buffer where trajectories are sampled as overlapping subsequences of length 32. Hidden states are initialized to zero at the start of each subsequence to avoid stale state issues common in off-policy learning.
Baselines: The recurrent approach is compared against standard feed-forward TD3 and Soft Actor-Critic (SAC), as well as a recurrent PPO (rPPO) variant.

Key Contributions

Benchmark Evaluation: The paper evaluates recurrent and non-recurrent deep RL methods on the AhnChemoEnv POMDP benchmark with fixed PK/PD parameters, effectively isolating the impact of partial observability.
Systematic Comparison: It provides a direct comparison of memory-based versus memoryless policies under both fully and partially observable conditions, extending prior work that focused only on full observability.
Empirical Robustness: The study demonstrates that memory-based policies are significantly more robust to hidden-state uncertainty and observation noise, reducing cross-seed variance by more than an order of magnitude compared to memoryless baselines.
Trajectory Analysis: The authors provide granular analysis showing that recurrent policies yield more consistent tumor suppression, stronger normal-cell recovery, and more structured dosing behaviors.

Results
Experiments were conducted over 10 random seeds with 400,000 environment steps per run.

Full Observability: Under full observability, all methods (rTD3, TD3, SAC) achieved competitive performance, with recurrence offering only modest benefits.
Partial Observability: The performance gap widened significantly.
- rTD3 achieved the strongest and most stable performance (Mean Return: $380.4 \pm 12.2$ ).
- Memoryless baselines (TD3, SAC) suffered severe degradation, exhibiting large oscillations, high variance, and negative mean returns (TD3: $-49.8 \pm 252.2$ ; SAC: $-154.1 \pm 219.7$ ).
- rPPO also maintained a positive return with lower variance than memoryless methods, though it was less performant than rTD3.
Behavioral Analysis: Trajectory plots revealed that rTD3 drove tumor populations to zero more consistently and enabled better recovery of normal cells compared to TD3. The recurrent policy exhibited more structured dosing (higher early concentration followed by synchronized tapering) and more stable immune responses.

Significance and Claims
The paper concludes that while recurrence offers limited advantage when the full state is known, it is essential for chemotherapy control under partial observability. Memory-based policies allow agents to infer latent physiological states (such as the hidden normal cell count) from noisy observation histories, leading to more reliable decision-making.

The authors highlight the practical significance of the off-policy recurrent architecture (rTD3). Since clinical deployment often relies on learning from historical patient records (offline learning) rather than active, potentially unsafe exploration, the ability of rTD3 to learn robustly from static datasets is a critical feature. The study acknowledges limitations, noting that the evaluation is entirely in silico with fixed PK/PD parameters and lacks validation on real patient data. Future work is proposed to incorporate patient-specific variability and explicit safety constraints.