💻 bioinformatics

Calibrated Uncertainty Quantification for Patient-Level AML Drug Sensitivity Prediction Using Split Conformal Prediction

This study introduces a split conformal prediction framework applied to the BeatAML 2.0 cohort that successfully generates statistically calibrated uncertainty intervals for patient-level AML drug sensitivity predictions, revealing distinct uncertainty patterns across drug classes while demonstrating that such uncertainty is independent of standard molecular risk classifications.

Original authors: Shokrzadeh, A. J., Shokrzadeh, P.

Published 2026-06-11

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Shokrzadeh, A. J., Shokrzadeh, P.

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine you are a chef trying to predict how a specific customer will react to a new dish. You have a massive cookbook of recipes (the drug compounds) and a library of notes on thousands of different diners' tastes (the patients' genetic data). Your goal is to tell a specific diner, "If you eat this dish, you will likely enjoy it this much."

The problem is, sometimes you might be wrong. In the past, computer models trying to make these predictions for blood cancer (AML) patients would just give a guess and maybe say, "I'm pretty sure," without actually knowing how sure they were. It was like a weather forecaster saying, "It might rain," without ever telling you the actual percentage chance.

This paper introduces a new, more honest way to make those predictions using a method called Split Conformal Prediction. Think of this method as a "safety net" or a "confidence ring" that the computer draws around every single guess.

Here is how the study worked, broken down into simple parts:

1. The Training Ground
The researchers used data from 318 real AML patients and tested how their cells reacted to 122 different drugs. They taught two different computer "brains" (one called Elastic Net and the other XGBoost) to look at a patient's genetic recipe (RNA-seq data) and predict how well a drug would work.

2. The "Safety Net" (Conformal Prediction)
Instead of just giving a single number (e.g., "This drug will work 70%"), the new method wraps that number in a range.

The Analogy: Imagine you are throwing darts at a target. A standard prediction is just saying, "The dart will hit the bullseye." The new method says, "The dart will hit somewhere inside this circle."
The Result: They tested three sizes of circles: small, medium, and large. They promised that if they said "90% of the time the dart hits the circle," it would actually happen 90% of the time. And guess what? It did. The computer was incredibly honest: when it promised 90% certainty, it was right 90.7% of the time. This is called being "calibrated."

3. What the "Safety Net" Revealed
By looking at how wide these safety circles were, the researchers found some interesting patterns:

Some drugs are harder to predict than others. For certain types of drugs (like HDAC and BCL-2 inhibitors), the safety circles were very wide. This means the computer was very unsure about the outcome. For other drugs (like MDM2 inhibitors), the circles were tighter, meaning the computer felt more confident.
The "Why" is a Mystery: They found that the patients' general risk level (how aggressive their cancer was) or a specific gene mutation (NPM1) didn't seem to make the predictions harder or easier. The uncertainty seemed to depend more on the type of drug being used rather than the specific patient's risk category.

4. The Big Takeaway
The main achievement of this paper is that they built a system that doesn't just guess; it tells you exactly how much it trusts its own guess. Before this, no one had applied this specific "safety net" math to real patient data in the BeatAML study.

In short, they didn't just build a better crystal ball; they built a crystal ball that tells you, "I'm 95% sure this is the answer," and proved that when it says that, it is actually right 95% of the time. This helps doctors understand not just what might happen, but how reliable that prediction is, even when dealing with the messy, complex biology of human cancer.

Technical Summary: Calibrated Uncertainty Quantification for Patient-Level AML Drug Sensitivity Prediction

Problem Statement
The accurate prediction of ex vivo drug sensitivity in acute myeloid leukemia (AML) patients using transcriptomic data remains a critical challenge in precision oncology. While existing computational approaches have explored uncertainty quantification in cancer drug response, these efforts have largely been confined to cell line data. In contrast, patient-level AML models typically depend on heuristic confidence measures rather than statistically calibrated uncertainty estimates. This gap limits the reliability of predictions in clinical settings where biological heterogeneity is substantial.

Methodology
This study introduces a framework that applies split conformal prediction to patient-level AML drug response modeling, utilizing the BeatAML 2.0 cohort. The methodology involved the following steps:

Data Scope: The analysis encompassed 318 AML patients, generating 34,764 patient-drug observations across 122 distinct compounds using bulk RNA-seq gene expression profiles.
Base Models: Two regression algorithms were trained on the transcriptomic data: Elastic Net and XGBoost.
Uncertainty Quantification: These baseline models were wrapped with split conformal prediction to generate statistically valid prediction intervals. This approach was evaluated across three nominal confidence levels (80%, 90%, and 95%).
Stratified Analysis: The study investigated prediction interval widths to identify patterns related to drug classes, molecular risk classifications (ELN2017), and specific mutation statuses (NPM1).

Key Results

Predictive Performance: The baseline models demonstrated moderate predictive capability, achieving median Pearson correlation coefficients ( $R$ ) of 0.291 for Elastic Net and 0.281 for XGBoost across the 122 drugs.
Calibration Accuracy: The application of split conformal prediction resulted in well-calibrated prediction intervals. The empirical coverages closely matched the nominal targets:
- 80% target $\rightarrow$ 81.4% empirical coverage
- 90% target $\rightarrow$ 90.7% empirical coverage
- 95% target $\rightarrow$ 95.5% empirical coverage
Drug-Class Specificity: Analysis of prediction interval widths revealed significant heterogeneity based on drug mechanism of action. HDAC and BCL-2 inhibitors exhibited markedly higher uncertainty compared to MDM2 inhibitors, suggesting a potential link between transcriptomic predictability and drug mechanism, though the study notes that some drug classes were represented by a limited number of compounds.
Biological Associations: Predictive uncertainty was found to be not significantly associated with established molecular risk factors. Specifically, no significant correlation was observed with ELN2017 molecular risk classification (Kruskal-Wallis $p=0.395$ ) or NPM1 mutation status ( $p=0.788$ ).

Significance and Claims
The paper asserts that statistically valid uncertainty quantification can be successfully achieved for patient-level AML drug response prediction, even in the presence of substantial biological heterogeneity. To the best of the authors' knowledge, this is the first published study to apply split conformal prediction to patient-level ex vivo drug sensitivity prediction within the BeatAML cohort.

The primary contribution of this work is the provision of a principled alternative to heuristic confidence scoring approaches. By demonstrating that conformal prediction yields well-calibrated intervals, the study offers a more rigorous framework for assessing prediction reliability in precision oncology, moving beyond uncalibrated confidence measures to provide statistically grounded uncertainty estimates.

Technical Summary: Calibrated Uncertainty Quantification for Patient-Level AML Drug Sensitivity Prediction

More like this