💻 computer science

Parkinson's Disease Drug Dose Optimization Using Multi-Objective Reinforcement Learning

This paper proposes a multi-objective reinforcement learning framework for optimizing Parkinson's disease medication dosing that generates a Pareto set of policies to explicitly model and navigate the trade-offs between motor symptom control and dyskinesia risks, outperforming clinician and random baselines while supporting transparent, patient-centered decision-making.

Original authors: Ahmad Rezaie Mianroodi, Seyed-Mohammad Fereshtehnejad, Frank Rudzicz

Published 2026-08-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ahmad Rezaie Mianroodi, Seyed-Mohammad Fereshtehnejad, Frank Rudzicz

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the captain of a spaceship navigating a tricky asteroid field. Your mission has two goals that often fight each other: you want to move as fast as possible to reach your destination, but you also want to avoid crashing into the asteroids. If you go too fast, you might crash; if you go too slow to be safe, you might never arrive. In the world of medicine, doctors face a similar "asteroid field" every day when treating patients with chronic diseases. They have to balance making a patient feel better right now against causing new, unwanted side effects later. For a long time, computer programs designed to help doctors make these decisions were like autopilots that could only follow one single rule, like "go as fast as possible." This forced the computer to decide for everyone exactly how much speed was worth the risk of a crash, ignoring the fact that different captains (or patients) might want different things. This new study dives into the complex world of Parkinson's disease, a condition that affects how people move, and asks a better question: Can we build a computer assistant that shows us all the possible ways to balance speed and safety, letting the doctor and patient choose the path that fits them best?

The researchers behind this study tackled the tricky job of managing medication for Parkinson's disease using a clever type of computer learning called "Multi-Objective Reinforcement Learning." Think of this as teaching a video game character not just to win, but to win in different styles. In this game, the character is a doctor adjusting the daily dose of medicine for a patient. The "score" isn't just one number; it's a two-part scorecard. One part measures how well the patient's muscles are working (motor symptoms), and the other measures how much the patient is experiencing involuntary shaking or jerking (dyskinesia), which is a common side effect of the medicine. The goal is to keep the muscles working well without making the shaking worse.

Instead of forcing the computer to pick a single "best" way to balance these two goals, the team used a method that maps out a whole "menu" of options. They fed the computer data from 823 real patients, tracking them over time to see how their symptoms changed with different medicine doses. The computer then built a map of 20 different "states" or situations a patient could be in, based on things like their age, how severe their symptoms were, and their cognitive function. For each of these situations, the computer didn't just give one answer; it drew a line of possibilities, called a "Pareto frontier."

Imagine this frontier as a spectrum of strategies. On one end, there are strategies that prioritize making the patient's muscles work perfectly, even if it means they might shake a little more. On the other end, there are strategies that prioritize keeping the shaking to a minimum, even if the muscles aren't quite as strong. In the middle, there are balanced approaches. The study found that these computer-generated strategies were generally better at managing the motor symptoms than the average decisions made by human doctors in the data, or random guesses. However, the trade-off was real: the strategies that were best at stopping the shaking were different from the ones that were best at improving movement.

The most exciting part is what this means for the future of care. The study suggests that instead of a computer telling a patient, "Take this much medicine because it's the mathematically perfect amount," the system could show a doctor and patient a range of choices. They could look at the map and say, "We are willing to accept a little more shaking to get better movement," or "We want to avoid shaking at all costs, even if movement is slightly slower." The researchers tested 500 different versions of their data to make sure their findings were solid, and the results consistently showed that this multi-goal approach could find better solutions than traditional methods.

However, it is important to remember that this was a simulation based on past data, not a test on real patients in a hospital. The computer learned from records of what happened in the past, so while it suggests these strategies would work, it hasn't proven they do work in the real world yet. The study explicitly rules out the idea that there is one single "perfect" dose for everyone. Instead, it argues that the best treatment depends on what the individual patient values most. By separating the math of the trade-offs from the choice of what to prioritize, this approach offers a new way to think about treatment—one that respects the unique preferences of every person living with Parkinson's.

Technical Summary: Parkinson's Disease Drug Dose Optimization Using Multi-Objective Reinforcement Learning

Problem Statement
Parkinson's disease (PD) management requires balancing the control of motor symptoms against adverse effects, particularly dyskinesia. This constitutes a complex sequential decision problem where clinicians must weigh short-term motor improvement against long-term complications. Existing reinforcement learning (RL) approaches in healthcare typically optimize a single scalar reward, necessitating the pre-specification of outcome priorities. This scalarization obscures clinically meaningful trade-offs and imposes fixed value judgments that fail to account for individual patient preferences or the dynamic nature of treatment goals. The authors posit that PD medication management is a natural candidate for Multi-Objective Reinforcement Learning (MORL), which can explicitly model and optimize competing objectives without collapsing them into a single weighted metric.

Methodology
The study formulates PD medication dose adjustment as a finite-horizon Multi-Objective Markov Decision Process (MOMDP) and applies MORL to longitudinal data from the Parkinson's Progression Markers Initiative (PPMI).

Data and Cohort: The analysis utilized data from 823 unique patients (3,726 visits) after cleaning. The dataset included clinical assessments such as MDS-UPDRS Part III (motor score), dyskinesia severity (NP4DYSKI), activities of daily living, cognitive scores (MoCA), and medication dosages (Levodopa Equivalent Daily Dose, LEDD).
State Space Construction: To ensure statistical reliability and clinical interpretability, the authors employed a decision-tree state aggregation method. A multi-output decision tree mapped continuous patient features (e.g., age, motor subtype, cognitive function, prior LEDD adjustments) into 20 discrete disease states. This ensured that each state contained sufficient samples for reliable estimation of transition and reward dynamics (constrained by a minimum sample size, e.g., $N_{min} = 100$ ) while retaining information relevant to all objectives.
Action Space: Actions were defined as discrete LEDT adjustments categorized into six clinically meaningful levels: No change, Decrease, and increases of up to 25%, 25–50%, 50–100%, and >100%.
Reward Function: The system optimizes a vector-valued reward $R \in \mathbb{R}^2$ $R \in R^{2}$ consisting of:
1. Motor symptom severity (negated MDS-UPDRS-III score).
2. Dyskinesia severity (negated binary indicator of dyskinesia presence).
  Both dimensions were normalized to the interval $[-100, 0]$ to prevent numerical dominance during optimization.
Algorithm: The authors utilized Convex Hull Value Iteration (CHVI), a model-based planning algorithm. Unlike single-policy methods, CHVI estimates the full set of Pareto-optimal policies. It first learns an empirical transition model from the training data and then computes the Pareto frontier of value vectors, representing the set of achievable trade-offs between motor control and dyskinesia.
Evaluation: The study employed a patient-level split (80% training, 20% testing) repeated over 500 bootstrap resamples to prevent data leakage. Policies were evaluated using stepwise Weighted Importance Sampling (WIS) on the held-out test set. Baselines included a random policy and a clinician behavior policy estimated via TD(0) from observed trajectories.

Key Results

Pareto-Optimal Policies: The framework successfully identified a spectrum of Pareto-optimal policies. Across 500 resamples, AI-derived Pareto-optimal policies demonstrated higher estimated motor-objective values compared to both clinician and random baselines ( $p < 0.001$ ).
Trade-off Dynamics: While motor performance was consistently superior to baselines across various preference weightings, estimated dyskinesia outcomes varied significantly depending on the specific objective weighting. Policies prioritizing motor symptoms tended to recommend larger dose increases (e.g., 50–100% or >100%), whereas policies prioritizing dyskinesia reduction shifted toward more conservative adjustments (e.g., no change or $\le$ 25% increase).
State Heterogeneity: The decision-tree partitioning identified 20 distinct disease states with heterogeneous clinical profiles. The learned policies exhibited state-dependent behaviors; for instance, in tremor-dominant states, the model tended to recommend larger dose increases, aligning with clinical literature suggesting these phenotypes may have lower susceptibility to levodopa-induced dyskinesia.
Alternative Objectives: When the secondary objective was changed to UPDRS Part IV or medication burden (negated LEDD), the results varied. Notably, when optimizing for motor outcomes and medication burden, the Pareto set collapsed to a single non-dominated solution, suggesting these specific objectives were aligned rather than conflicting in the learned model.

Significance and Claims
The paper claims to be the first application of MORL to a chronic neurological disorder. Its primary contribution is methodological: it provides a framework for studying preference-sensitive sequential treatment decisions by separating outcome estimation from value prioritization.

Explicit Trade-offs: The approach makes treatment trade-offs explicit, allowing clinicians and patients to select from a frontier of policies based on individual priorities rather than deferring to a fixed algorithmic prescription.
Shared Decision-Making: By operationalizing shared decision-making, the framework supports transparent deliberation over competing priorities (motor control vs. dyskinesia risk).
Modest Scope of Claims: The authors explicitly state that the results are hypothesis-generating and evaluation-dependent. They do not claim real-world clinical benefit or causal treatment effects. The findings are limited to offline evaluation on observational data, and the authors caution that prospective validation, sensitivity analyses, and evaluation in richer cohorts are necessary before the framework can be considered for clinical decision support. The study concludes that MORL is a useful tool for exploring the structure of treatment trade-offs in PD, rather than a validated dosing recommendation system.

Technical Summary: Parkinson's Disease Drug Dose Optimization Using Multi-Objective Reinforcement Learning

More like this