Beyond AUROC: External Validation, Calibration, and Explainability of Machine Learning Models for 30-Day Hospital Readmission Using Electronic Health Records—A Rapid Systematic Review

Every year, millions of patients leave a hospital only to return within a month, often because their recovery stalled or their care plan fell apart. This cycle of readmission is a major burden on healthcare systems, driving up costs and signaling that something went wrong with the transition from hospital to home. To stop this, hospitals have turned to machine learning, a branch of computer science where software learns to find patterns in vast amounts of data. By feeding these programs electronic health records—digital files containing lab results, medication lists, and doctor's notes—researchers hope to build tools that can spot the patients most likely to return and send them extra help before they leave. For years, the scientific community has judged these tools by a single number: a score that measures how well the model ranks patients from lowest risk to highest risk. If a model can correctly say "this patient is riskier than that one," it gets a high score, and that has been enough to declare it a success.

However, a new analysis suggests that this single score tells only half the story. A rapid systematic review conducted by researchers at Chittagong University of Engineering and Technology examined 156 studies published between 2020 and 2026 that used machine learning to predict 30-day hospital readmissions. The team found that while these models often look impressive on paper, they frequently fail to prove they will work in the real world. The researchers discovered that almost all the studies relied on internal testing, where the model is checked against data from the same hospital that created it. Very few studies tested whether the model would work at a different hospital or with data from a later time period. Furthermore, the studies rarely checked if the risk percentages the models produced were actually accurate, or if the reasons the models gave for their predictions made sense to human doctors. The review concludes that the field has moved too fast in adopting flashy new features while neglecting the basic checks needed to ensure these tools are safe and reliable for patients.

The core issue identified is that a model can be good at ranking patients without being good at predicting actual outcomes. Imagine a weather app that correctly tells you that Tuesday is more likely to rain than Monday, but fails to tell you that the chance of rain is actually zero percent on both days. In the context of hospital readmissions, this distinction is critical. Hospitals need to know the actual probability of a patient returning so they can decide how many resources to allocate. If a model predicts a 90 percent chance of readmission for a patient who only has a 10 percent chance, the hospital might waste scarce transitional care services on that patient while missing others who truly need them. The review found that while 85 percent of the studies reported the ranking score, only about 15 percent mentioned checking if their predictions matched reality, and only one study used a specific metric designed to measure this accuracy.

Beyond accuracy, the researchers looked at whether these models could travel. A model built on data from a large urban teaching hospital in one city might fail completely when applied to a rural clinic in another state, simply because the patient populations and medical practices are different. This is known as external validation. The review found that only 12 percent of the studies tested their models in a new setting or at a later time. When the few studies that did perform these tests shared their results, the models almost always performed worse in the new environment. In five cases where researchers compared the model's performance in its home hospital versus a new one, the ability to correctly rank patients dropped in every single instance. This suggests that many models are overfitted to their specific local data, capturing quirks of one hospital rather than universal signs of risk.

Another major gap in the research is explainability. As machine learning models have become more complex, researchers have started using tools to explain why a model made a specific prediction, often highlighting which factors, like high blood pressure or a specific medication, contributed most to the risk score. The review noted that the use of these explanation tools has surged, jumping from less than 6 percent of studies in the early years of the review period to nearly 30 percent in the most recent years. However, the researchers found that while studies were quick to show these explanations, they almost never tested if the explanations were actually correct or stable. A model might point to a specific lab value as the reason for a high risk, but if that reason changes randomly when the data is slightly tweaked, the explanation is not trustworthy. The review found no abstracts that evaluated whether the explanations provided were reliable or if doctors found them useful.

The study also examined whether these models treated different groups of people fairly. Readmission rates can be influenced by factors outside of pure biology, such as access to transportation or the ability to afford medication. If a model learns from historical data that reflects these inequalities, it might unfairly flag certain groups as high risk simply because they have less access to care, rather than because they are medically sicker. The review found that only a tiny fraction of studies, about 4.5 percent, checked for these disparities across different groups of people based on race, gender, or age. This lack of scrutiny means that hospitals could inadvertently deploy tools that reinforce existing biases in the healthcare system.

The researchers concluded that the field is prioritizing the wrong things. The rapid rise in the use of complex explanation tools has outpaced the adoption of fundamental safety checks like testing models in new locations or verifying their accuracy. The authors argue that before these models are used to make real decisions about patient care, they must pass a minimum set of standards. These standards include testing the model on data from a different hospital or time period, checking that the predicted risks match the actual outcomes, and ensuring the model does not discriminate against specific groups. The review suggests that the next breakthrough in this field will not come from inventing more complex algorithms, but from doing a better job of proving that the tools we already have are safe, accurate, and fair for everyone.