Benchmarking machine learning for cardiovascular disease classification across 46 countries with geographic and temporal transportability assessment
This study benchmarks machine learning models for cardiovascular disease classification across 46 countries using WHO STEPS data, revealing that complex algorithms offer only marginal performance gains over logistic regression while highlighting substantial geographic and temporal heterogeneity that necessitates prioritizing transportability, calibration, and minority-class performance over pooled ROC AUC in public health AI.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Heart disease remains the leading cause of death worldwide, claiming millions of lives every year. In many parts of the world, the burden is even heavier, with more than three-quarters of these deaths occurring in low- and middle-income countries. To fight this, health officials rely on large surveys that ask people about their health, their habits, and their access to medical care. These surveys gather data on things like blood pressure, smoking, and diet, which are known to influence heart health. The hope is that by analyzing this information, computers can learn to spot patterns that predict who is at risk, allowing doctors to intervene earlier.
In recent years, there has been a rush to use advanced computer programs, known as machine learning, to find these patterns. These programs are often touted as being able to solve problems that simpler methods cannot, finding hidden connections in vast amounts of data. However, a major question remains: do these complex programs actually work better when applied to real-world situations across different countries and over time? Often, these tools are tested in ways that make them look more successful than they truly are, such as by splitting data from a single country into training and testing groups. This approach can hide the fact that a model might fail completely when faced with a new country, a different year, or a population with different access to healthcare.
A new study set out to test these computer models against a much harder challenge. Researchers gathered health survey data from 46 different countries, covering nearly 188,000 adults. They asked a straightforward question: can a computer program accurately identify people who already have heart disease based on their survey answers, and will that program still work if it is sent to a country it has never seen before? To find the answer, they compared the performance of several different types of computer models, ranging from simple, transparent statistical methods to complex, modern artificial intelligence systems.
The researchers did not just look for the highest overall score. Instead, they designed a rigorous test that mimicked real-world deployment. They trained the models on data from most of the countries and then tested them on entire countries that were completely left out of the training process. They also checked how well the models held up over time by training them on older survey data and testing them on newer data from the same countries. They paid close attention to how well the models handled the fact that heart disease is relatively rare in these surveys, meaning that a model could be wrong most of the time and still look good on a standard score. They also examined whether the models were simply learning to recognize people who had already visited a doctor for other conditions, rather than identifying the underlying risk of heart disease itself.
The results offered a sobering reality check for the field of medical artificial intelligence. The most complex computer models did not perform significantly better than a simple, well-understood statistical method. In fact, the difference in their ability to correctly identify heart disease cases was so small that it was barely noticeable. While the complex models achieved a tiny edge in some specific measurements, they did not offer the dramatic improvement that many had hoped for. More importantly, the study found that the performance of these models varied wildly depending on where they were used. A model that worked reasonably well in one region might fail almost completely in another. For example, in some parts of Central Asia, the models were quite accurate, but in several Pacific island nations and parts of Africa, their ability to distinguish between those with and without heart disease dropped to near-random levels.
The study also revealed that these models are not stable over time. When the researchers trained a model on data from a few years ago and tested it on data from a later year, the results were often poor. The models struggled to adapt to changes in the population, shifts in how diseases were diagnosed, or improvements in healthcare access that occurred between survey waves. This suggests that a model built today might not be reliable tomorrow without significant adjustments. Furthermore, the researchers found that the models relied heavily on whether a person had been diagnosed with high blood pressure or diabetes. When they removed these specific pieces of information from the data, the models' performance dropped noticeably. This indicates that the computers were partly learning to identify people who had already interacted with the healthcare system, rather than purely identifying the biological risk of heart disease. In countries where many people do not have regular access to doctors, this reliance could lead to the models missing the very people who need help the most.
The researchers concluded that the choice of a complex algorithm is less important than the context in which it is used. The differences between countries and over time were far greater than the differences between the computer models themselves. They argued that before any such tool is used to guide public health decisions, it must be tested not just for its overall score, but for its ability to work in specific local settings, its reliability over time, and its fairness across different groups of people. The study suggests that for global health surveillance, transparency and robustness are more valuable than complexity. A simple model that is well-understood and can be easily adjusted for local conditions may be a far better tool for saving lives than a sophisticated black box that fails when the environment changes. The path forward, the authors suggest, is not to chase the most powerful computer program, but to build systems that are honest about their limitations and adaptable to the diverse realities of the world they are meant to serve.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.