Methods for Development and Validation of Individual- Level Clinical Prediction Models for Multi-State Processes: A Methodological Scoping Review
This methodological scoping review systematically summarizes 105 studies on the development and validation of individual-level clinical prediction models for multi-state processes, highlighting substantial progress in model specification for Markov processes while identifying critical gaps in areas such as sample size determination, missing data handling, and comprehensive validation.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the world of medicine, predicting the future is a constant challenge. Doctors often rely on tools that estimate the chance of a single event happening, such as the likelihood of a heart attack within the next decade. These tools take a patient's current information—like age, weight, or smoking habits—and calculate a risk score. However, human health is rarely a straight line. Patients often move through a series of different health states over time. A person might be healthy, then develop a chronic condition, then suffer a complication, and finally reach a terminal state. Sometimes, these events compete with one another; a patient might die from one cause before another condition has a chance to develop. To understand these complex journeys, scientists use a framework called a multi-state model. Think of it as a map that tracks not just a single destination, but the entire path a patient travels, including the stops they make and the order in which they arrive. The goal is to use this map to predict an individual's specific future: not just if they will get sick, but what state they will be in next, and how long they might stay there.
A team of researchers at the University of Manchester recently set out to examine how well the scientific community is building these predictive maps for individual patients. They conducted a comprehensive review of existing studies to see what methods are available for creating and testing these complex models. The researchers searched through thousands of academic papers, filtering out those that did not focus on individual risk or that dealt only with specific types of data like spatial patterns. After a rigorous screening process, they identified 105 studies that offered new ways to build or test these multi-state prediction models. Their investigation revealed a landscape that is strong in some areas but surprisingly quiet in others.
The review found that the majority of the work focuses on a specific type of model known as a Markov process. In this approach, the prediction of what happens next depends only on the patient's current state, ignoring how long they have been there or what path they took to get there. The researchers found 55 papers dedicated to developing methods for these Markov models. These studies offered various techniques to handle the math, including ways to select the most important patient factors and methods to fit the models to data. There were also 18 papers exploring semi-Markov models, where the time spent in a current state matters, and 14 papers on non-Markov models, which account for the full history of a patient's journey. A smaller number of studies, 11 in total, proposed methods that do not rely on any specific assumption about how the process works, while seven papers focused on a technique called pseudo-values, which estimates risks at specific future times without modeling the entire journey step-by-step.
Despite this progress in building the models, the researchers discovered significant gaps in the rest of the pipeline. While scientists have developed many ways to construct these complex models, very few studies have addressed how to determine the right amount of data needed to build them reliably. None of the 105 development-focused papers offered a method for calculating the necessary sample size for multi-state predictions. Similarly, only two papers tackled the common problem of missing data, where patient information is incomplete. The review also highlighted a lack of guidance on how to handle missing information in the complex sequence of events that define these models.
Perhaps the most striking finding was the scarcity of work on validation. In the world of prediction, validation is the process of checking if a model actually works in the real world. The researchers found only four studies that focused on methods for validating these multi-state models. These few papers looked at how well the models could distinguish between high-risk and low-risk patients, or how accurately their predicted probabilities matched reality. The authors noted that while we have many tools to build the models, we lack robust ways to test them, especially when it comes to assessing their performance across multiple different health states and over time.
The review concludes that while the mathematical machinery for describing these complex health journeys has advanced, the practical tools needed to ensure these models are safe and reliable for clinical use are still underdeveloped. The researchers suggest that future work must prioritize filling these gaps. Scientists need to create better ways to determine how much data is required, how to handle missing information, and, most critically, how to rigorously test these models before they are used to guide patient care. Until these pieces are in place, the promise of using multi-state models to provide precise, individualized predictions for complex diseases remains only partially realized.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.