Prospective validation of an uncertainty-aware machine learning model for secondary progressive multiple sclerosis classification

Multiple sclerosis is a condition where the body's immune system mistakenly attacks the protective covering of nerve fibers, disrupting communication between the brain and the rest of the body. For many people, the disease begins with a relapsing-remitting pattern, where symptoms flare up and then fade away. Over time, however, the disease can shift into a secondary progressive phase, where disability steadily worsens without these distinct periods of recovery. Identifying exactly when this shift happens is notoriously difficult for doctors. Unlike the initial diagnosis, which relies on specific, measurable tests, recognizing this later stage often depends on a doctor's judgment and a long history of patient visits. This uncertainty can delay the start of treatments that are specifically designed for the progressive stage, leaving patients without the right care for years.

A team of researchers in Sweden has developed a new way to help spot this transition earlier and more reliably. They built a computer model that learns from the medical records of thousands of patients to recognize the subtle signs that a person is moving from the relapsing stage to the progressive one. What makes this work different is not just that the model can make a prediction, but that it can also tell a doctor how sure it is about that prediction. In a recent study, the researchers tested this model on fresh data collected after the model was built, proving that it works on new patients it has never seen before. They found that the model could identify the shift to the progressive stage much more often than the current standard method used by doctors, while still providing a clear measure of its own confidence.

The researchers started by teaching their computer model using a massive database of medical records from the Swedish Multiple Sclerosis Registry. This database contained information on over ten thousand people and nearly ninety thousand hospital visits. The model learned to look at specific details from each visit, such as the patient's age, their disability score, the types of medications they were taking, and results from brain scans. It learned to distinguish between patients who were still in the relapsing phase and those who had moved into the progressive phase, based on how doctors had labeled them in the registry. Crucially, the model was designed to include a system that calculates uncertainty. This means that when the model makes a guess, it also provides a confidence score, letting the user know if the prediction is a strong certainty or if the data is too ambiguous to be sure.

To see if this model would work in the real world, the team did not just test it on the same data they used to teach it. Instead, they waited for new data to arrive. They took records from 2025, which included over fifteen thousand individuals and forty-five thousand hospital visits that were completely new to the model. This approach is known as prospective validation, and it is a rigorous way to ensure a tool works on future patients, not just past ones. When they ran the new data through the model, it performed with high accuracy. At the level of individual hospital visits, the model correctly identified the relapsing stage in 92 percent of cases and the progressive stage in 95 percent of cases. More importantly, the model remained well-calibrated, meaning its confidence scores matched reality; when it said it was 93 percent sure, it was right about 93 percent of the time.

The researchers then looked at how well the model could identify individual patients who had transitioned to the progressive stage. They found that the model successfully identified 98 percent of the patients who had actually transitioned, a rate significantly higher than the current standard method, which only caught 71 percent of them. The standard method, known as the Lorscheider criteria, relies on a strict set of rules requiring a patient to show a steady decline in function over a long period without any recent relapses. While this method is very specific and rarely makes false alarms, it misses many cases because it requires a long wait to confirm the change. The new model, by contrast, could spot the transition much earlier, often at the exact visit when a doctor first noted the change, or even before.

One of the most valuable aspects of this work is that the model does not just give a yes-or-no answer; it explains itself. By using a technique that breaks down which factors influenced its decision, the researchers showed that the model was relying on the same clinical signs that doctors look for, such as disability scores and age. This transparency helps build trust, as it shows the model is not finding hidden patterns that make no medical sense. Furthermore, the model's ability to say "I am not sure" is a safety feature. In cases where the data was confusing, the model would flag the uncertainty rather than making a risky guess, allowing a human doctor to step in and review the case more closely.

The study also highlighted a key difference in how the two methods handle the transition period. The model identified 90 percent of the patients who were transitioning, whereas the standard criteria only identified 57 percent. This suggests that the standard rules are too strict and may be leaving many patients without the specific treatments available for the progressive stage. With new medications recently approved that target the progressive phase specifically, missing these patients means they are not getting the care they need. The model offers a way to catch these cases earlier, potentially allowing doctors to start the right treatment sooner.

However, the researchers are careful to note that their model is not a replacement for a doctor. It is a tool designed to support clinical decision-making. The study was conducted using data from Sweden, and the researchers acknowledge that the model would need to be tested in other countries with different healthcare systems and patient populations before it could be used globally. They also point out that the model learns from how doctors currently label patients, so if those labels are delayed or inconsistent, the model will reflect that. Despite these limitations, the study demonstrates that a computer model trained on real-world data can reliably predict disease progression and provide a measure of confidence that is currently missing from standard practice.

The ultimate goal of this research is to bridge the gap between the moment a patient's condition changes and the moment they receive the appropriate treatment. By providing a reliable, early warning system that comes with a clear statement of confidence, this tool could help ensure that patients are not left waiting for years to be recognized as having the progressive form of the disease. The study shows that with the right approach, artificial intelligence can be integrated into medicine not as a mysterious black box, but as a transparent, calibrated assistant that helps doctors see the future of a patient's health with greater clarity.