Temporally validated clinical trial outcome prediction from public registry text: a leakage-audited comparison of biomedical transformers and classical NLP

Developing new medicines is a long, expensive, and uncertain journey. Most drug candidates that enter testing never make it to the pharmacy shelf. Because the stakes are so high, researchers and companies constantly look for ways to predict which trials are likely to succeed before they spend years and millions of dollars on them. For decades, these predictions have relied on complex data about the drug's molecular structure, the specific disease it targets, or the company running the study. Often, this information is locked away in private databases or requires expensive subscriptions to access. This creates a barrier where only well-funded organizations can build the tools needed to forecast the future of medical research. But what if the answers were already sitting in plain sight, written in the public descriptions of the studies themselves?

A recent study by independent researcher Michael Doane explored whether a computer program could learn to predict the success of clinical trials using only the free, public text found on the ClinicalTrials.gov website. This website serves as a global registry where every clinical trial must be registered, detailing the study's design, the patients involved, and the goals of the research. Doane wanted to know if a modern language model—a type of artificial intelligence trained to understand human language—could read these public descriptions and rank the trials by their likelihood of success. To do this, he used a massive dataset of trial outcomes generated by the Clinical Trial Outcome Dataset, which compiles evidence from news reports and public results to assign a probability of success to each trial. The goal was to see if an AI could learn from these automated probabilities and then apply that knowledge to new, unseen trials, specifically those focused on neuroscience, without ever seeing the final results during its training.

The researcher trained three separate computer models, one for each stage of drug testing: early safety checks, intermediate testing, and large-scale final trials. He fed these models the public text from thousands of trials completed before 2019, using the automated success probabilities as the only guide for what to learn. Crucially, the models were not allowed to see the trial results, the dates, or the specific identifiers that might give away the answer. Instead, they had to rely solely on the narrative descriptions of the study design and the people involved. To ensure the results were genuine and not a trick of the data, the researcher set a strict rule: the models would only be tested on trials that finished after 2019, a time period the models had never seen. This approach, known as temporal validation, mimics how a prediction tool would actually be used in the real world, looking forward into the future rather than just memorizing the past.

The results showed that the computer models could indeed find a signal for success within the public text. The primary model, based on a system called BioBERT which is pre-trained on biomedical literature, successfully ranked the trials. In the early safety phase, it achieved an AUC of 0.67. In the intermediate phase, this score rose to about 0.71, and in the final large-scale phase, it reached roughly 0.72. While these numbers might seem modest, they represent a significant ability to distinguish between successful and unsuccessful trials using only public information. The model performed better than random guessing and better than older, simpler statistical methods in the later stages of testing. Another model, called BiomedBERT, performed even slightly better in the final phase, reaching an AUC of nearly 0.75.

What makes this finding particularly notable is how efficiently it was achieved. The entire process, including training the complex AI models and running the comparisons, took only 3.5 hours on a single graphics processing unit, costing just over one dollar in electricity and cloud computing fees. This stands in stark contrast to the massive supercomputers and expensive datasets often associated with artificial intelligence research. The study demonstrated that a researcher with modest resources could reproduce these results using only free, public data. The models were able to learn from the text descriptions alone, proving that the way a trial is described in a public registry contains hidden patterns that correlate with its eventual outcome.

However, the study also highlighted important limitations. While the models were good at ranking trials—telling which ones were more likely to succeed than others—they were not perfect at predicting the exact probability of success. In the earlier phases, the models tended to be overly optimistic, predicting higher success rates than actually occurred. This suggests that while the text provides a useful signal for sorting trials, it cannot yet replace the need for human judgment or more detailed data. The researchers also noted that the text descriptions might change over time, and if a registry entry is updated after a trial finishes, it could accidentally leak the answer to the model. To prevent this, the study used strict rules to scrub any mention of results from the text before the models read it.

Ultimately, this work suggests that the public record of clinical trials holds more predictive power than previously realized. By using automated labels to teach a language model, the researcher showed that it is possible to build a tool that ranks drug trials based on their public descriptions alone. This approach does not require access to private corporate data or expensive proprietary databases. It offers a transparent, reproducible baseline that any researcher can use to screen trials or prioritize which studies deserve closer attention. While the models are not yet ready to make final decisions on drug development, they provide a powerful new way to look at the vast amount of information already available to the public, turning simple text into a guide for navigating the uncertain path of medical discovery.