💻 computer science

Leveraging Machine Learning Models to Predict Probability of Technical and Regulatory Success

This paper demonstrates that interpretable machine learning models trained on public ClinicalTrials.gov protocol features can accurately predict the probability of technical and regulatory success for small-molecule trials, outperforming standard benchmarks and offering a reproducible tool for risk-adjusted valuation.

Original authors: Mason Kim, Michael Doane, James Melican, Kevin Wang, Haewon Jung, Cedric Liu

Published 2026-07-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mason Kim, Michael Doane, James Melican, Kevin Wang, Haewon Jung, Cedric Liu

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a venture capitalist deciding whether to invest in a new startup. You know that most startups fail, but you want to know which ones have the best chance of succeeding. In the world of medicine, these "startups" are new drugs, and the "investors" are pharmaceutical companies.

The paper you provided is about building a crystal ball for these drug companies. However, instead of magic, they used math and public data to predict whether a new drug will survive the long, difficult journey to become an approved medicine.

Here is the breakdown of their work in simple terms:

1. The Problem: The "Black Box" of Drug Success

Developing a new drug is incredibly expensive and risky. On average, only about 1 in 10 new drugs makes it from the start of testing to final approval.

The Old Way: Companies usually guess the odds of success by looking at historical averages (e.g., "Drugs for heart disease have a 15% success rate") or by asking a group of experts to use their intuition.
The Issue: Historical averages are too blunt (they don't look at the specific drug), and expert opinions can be biased or too slow. Also, many high-tech prediction models used today are like "black boxes"—they give an answer, but no one knows how they got there, and they often require secret, expensive data that regular people can't see.

2. The Solution: A Transparent, Public "Weather Forecast"

The authors wanted to build a model that is:

Reproducible: Anyone can build it.
Interpretable: You can look under the hood and see exactly why it made a prediction.
Public: It only uses data available for free on the internet (specifically ClinicalTrials.gov, a giant public database of medical studies).

They treated the prediction like a weather forecast. Just as meteorologists use past weather patterns to predict rain, this model uses past clinical trial data to predict "drug rain" (success) or "drug drought" (failure).

3. How They Built the Model

The Ingredients: They downloaded information from thousands of past clinical trials. They didn't look at secret chemical formulas or private company reports. They only looked at the "protocol" (the rulebook) of the trial:
- Who is paying for it? (The Sponsor)
- What disease is it treating? (Therapeutic Area)
- What stage is the trial in? (Phase 1, 2, or 3)
- How is the trial designed? (Is it randomized? Who is the patient?)
The Training: They fed this data into a computer program (a "Random Forest" model, which is like a team of many decision-makers voting on the outcome). They taught the computer using trials that finished before 2017.
The Test: They then asked the computer to predict the outcome of trials that started after 2017, which it had never seen before. This is like testing a student on a new exam after teaching them with old practice tests.

4. The Results: Better Than the Average Guess

The computer model did a better job than the standard "average" guesses used by the industry.

The Score: The model achieved a score (called ROC-AUC) of 0.788. In the world of predictions, this is a solid "B+" or "A-," meaning it is significantly better than just flipping a coin or using a generic average.
The "Brier Score": This measures how calibrated the predictions are. If the model says there is a 70% chance of success, does it actually succeed 70% of the time? The model was very good at this, meaning its probabilities were trustworthy.

5. What the Model Learned (The "Why")

Because the model is transparent, the authors could ask it, "What factors mattered most?" It gave them three main answers, which align with common sense but confirm it with data:

The Sponsor Matters: Trials funded by big pharmaceutical companies were more likely to succeed than those funded by universities or the government. (Think of it as a well-funded startup vs. a garage project).
The Disease Matters: Trials for infectious diseases and metabolic issues had higher success rates, while cancer (oncology) trials were much harder to pass (lower success rates).
The Stage Matters: As a drug gets closer to the finish line (Phase 3), the odds of success go up. Phase 2 is the "danger zone" where many drugs fail.

6. The Catch (Limitations)

The authors are honest about the model's limits:

It's not perfect: It's not a magic 100% predictor. It still makes mistakes.
Time Gap: If you train the model on data from 10 years ago and try to predict today, it gets slightly less accurate, but it still works better than the old benchmarks.
Missing Info: The model didn't use the actual chemical structure of the drug or deep scientific text, only the public "rulebook" of the trial. Adding those details might make it even smarter.

The Bottom Line

This paper proves that you don't need secret, expensive data or complex, unexplainable AI to predict drug success. By using public data and simple, transparent math, you can build a tool that helps companies make smarter decisions about which drugs to invest in. It's like giving everyone a better map for navigating the risky journey of drug development.

Technical Summary: Leveraging Machine Learning Models to Predict Probability of Technical and Regulatory Success

Problem Statement

The pharmaceutical industry faces high failure rates in drug development, with an average probability of technical and regulatory success (pTRS) for investigational new drugs (INDs) at approximately 9.6%. Accurate pTRS assessment is critical for risk-adjusted valuation (rNPV), portfolio management, and strategic decision-making. However, existing assessment methods often rely on proprietary data, complex "black-box" architectures, or subjective expert elicitation, which can be difficult to replicate, calibrate, or integrate into standard workflows. There is a lack of standardized, reproducible frameworks that utilize only publicly available data to generate calibrated, interpretable pTRS estimates.

Methodology

Data Sources and Preprocessing

The study utilized two primary data sources:

Feature Space: Protocol-section data retrieved from ClinicalTrials.gov via API (as of January 2026). Features were extracted from eight modules: Identification, Sponsor/Collaborators, Description, Conditions, Design, Arm Interventions, Outcomes, and Eligibility.
Outcome Labels: Binary success/failure labels (0/1) for small-molecule trials through 2021, sourced from the HINT clinical-trial-outcome repository (derived from IQVIA data).

Data Cleaning and Engineering:

Cleaning: Non-text features (e.g., age, dates) were standardized numerically. Categorical lists (e.g., conditions) were converted to pipe-delimited strings.
Therapeutic Area Mapping: Using GPT-4o with few-shot prompting, trials were classified into one of 21 predefined therapeutic areas (e.g., Oncology, Cardiovascular, Infectious Disease) based on conditions and keywords.
Encoding: Categorical and multi-categorical variables were one-hot encoded.
Temporal Split: To simulate real-world forecasting, data was split temporally:
- Training Set: Trials with primary completion dates before January 1, 2017 ( $n=10,224$ ).
- Testing Set: Trials with start dates after January 1, 2017 ( $n=478$ ).
Class Imbalance: Class weights were applied inversely proportional to class frequencies to penalize misclassification of failures more severely.

Model Development

Three interpretable machine learning models were evaluated:

Logistic Regression: Used as a baseline.
Random Forest (RF): Evaluated with hyperparameter tuning via random search (20 combinations, 5-fold cross-validation).
XGBoost: Evaluated similarly to Random Forest.

Evaluation Metrics:

Discrimination: ROC-AUC and PR-AUC.
Calibration: Brier score (decomposed into reliability, resolution, and uncertainty).
Benchmark Comparison: An "Outperformance Rate" metric was defined to measure how often the model's predicted probability was closer to the true outcome than a derived benchmark (mean success rate by disease category and phase).
Interpretability: SHAP (SHapley Additive exPlanations) values were used to rank feature importance and explain individual predictions.

Key Results

Model Performance

The Random Forest model demonstrated the strongest aggregate performance across all metrics, outperforming the Logistic Regression baseline and standard benchmarks in most settings:

ROC-AUC: 0.788 (vs. 0.726 for baseline).
PR-AUC: 0.794 (vs. 0.748 for baseline).
Brier Score: 0.196 (vs. 0.220 for baseline), indicating superior calibration.
Note: The baseline model slightly outperformed Random Forest only in the PR-AUC metric for Phase 3 trials.

Temporal Stability

The study tested the impact of temporal gaps (0, 1, 3, and 5 years) between training and test data. While performance showed a modest, consistent decline as the gap increased (e.g., ROC-AUC dropped from 0.755 at 0-year gap to 0.748 at 5-year gap), the model remained robust and continued to outperform benchmarks even with a 5-year gap.

Feature Importance (SHAP Analysis)

The SHAP analysis identified the following as the dominant drivers of pTRS predictions:

Sponsor Type: Industry sponsorship was the strongest positive driver; non-industry sponsorship (e.g., NIH, academic) correlated with lower predicted success.
Therapeutic Area: Oncology was associated with lower pTRS, while Infectious Disease and Metabolic disorders showed higher pTRS.
Trial Phase: Phase 3 trials showed higher predicted success, while Phase 2 (proof-of-concept) showed lower success, consistent with historical attrition rates.
Design Variables: Randomized allocation, masking, and single-group assignments had moderate but consistent effects.

Significance and Claims

The paper claims that its primary contribution is practical and methodological rather than achieving state-of-the-art predictive accuracy through architectural complexity.

Reproducibility: The study demonstrates that calibrated, decision-useful pTRS estimates can be generated using only public data (ClinicalTrials.gov) and open labels, removing reliance on proprietary databases or complex deep learning pipelines (e.g., HINT, inClinico).
Interpretability: By using models like Random Forest and SHAP, the approach provides transparent explanations for stakeholders, facilitating integration into expert elicitation and valuation workflows where "black-box" models are often rejected.
Decision Utility: The authors argue that even modest improvements in probabilistic accuracy (e.g., 3–4 percentage points) are significant at the portfolio scale, influencing capital deployment and Go/No-Go decisions.
Benchmarking Strategy: The paper introduces a benchmarking approach derived from the training data itself (phase-by-therapeutic-area means) to create a fair, reproducible comparison metric that avoids the biases of external, potentially incomplete industry databases.

Limitations Acknowledged

The authors explicitly note several constraints:

Data Recency: Labels are limited to small-molecule trials through 2021; shifts in regulatory or trial conduct since then may affect forward performance.
Feature Scope: The model excludes rich free-text descriptions, molecular properties (SMILES), and PK/PD/toxicity data, which may cap performance, particularly for first-in-class or oncology assets.
LLM Categorization: Therapeutic area mapping relied on LLMs, which could introduce structured misclassification.
Benchmark Derivation: The "outperformance" metric relies on benchmarks derived from the training set; future work should validate against external benchmarks.

In conclusion, the paper posits that interpretable models trained on open clinical trial features offer a viable, practical foundation for risk quantification that complements, rather than replaces, expert judgment and traditional benchmarks.