💻 computer science

Comparative Evaluation of Machine Learning Models for Heart Disease Prediction: Performance, Calibration, Threshold Selection, and Explainability

This study evaluates five machine learning models on the Cleveland Heart Disease dataset, finding that logistic regression with a calibrated threshold of 0.552 achieved the best performance (85.71% accuracy and 0.939 ROC-AUC) while identifying major vessels, chest pain type, and thallium test results as key predictors, though the authors emphasize the need for external validation before generalizing these findings to diverse populations.

Original authors: Stanley Kirunda Dhakaba¹, Patience Atukunda¹, Anthony Bua²

Published 2026-10-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Stanley Kirunda Dhakaba¹, Patience Atukunda¹, Anthony Bua²

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Heart disease remains the world's leading cause of death, claiming millions of lives annually, with a disproportionate burden falling on low- and middle-income nations. In the modern era, doctors and researchers increasingly turn to machine learning to spot the subtle patterns in patient data that signal rising risk before a crisis occurs. These computer programs act as powerful tools for sifting through complex medical histories, looking for connections between factors like age, blood pressure, and cholesterol levels to predict who might develop heart disease. However, while these tools show great promise, their real-world value depends entirely on how they are tested. A model that performs well on a small, specific group of patients might fail completely when applied to a different population. To be truly useful, a prediction tool must not only guess correctly often enough but also provide reliable probabilities and work consistently across different types of people.

A team of researchers from universities in Uganda and Victoria set out to test five different machine learning approaches using a well-known collection of heart health records. Their goal was not just to find the single most accurate computer program, but to understand how these programs behave, how they make their decisions, and whether they remain reliable when the data changes slightly. They worked with a dataset containing 303 patient records, each describing a person's age, blood pressure, cholesterol, heart rate, and results from specific cardiac tests. The researchers split this data into two groups: a larger group to teach the computer programs what to look for, and a smaller, separate group to test how well the programs performed on patients they had never seen before. They ensured the split was fair, keeping the same proportion of sick and healthy patients in both groups.

After training the five different models, the researchers found that a method called logistic regression emerged as the most balanced and reliable performer. This approach correctly identified heart disease in about 85 out of every 100 patients in the test group. More importantly, it did so with a high degree of precision, meaning it rarely flagged healthy people as sick, and it was also very good at catching actual cases of the disease. While other models, such as random forests and decision trees, showed promise, they either made more mistakes in identifying sick patients or struggled to distinguish them from healthy ones as effectively. One model, known as Bayes, was excellent at finding sick patients but was too eager to sound the alarm, incorrectly labeling many healthy people as having heart disease. The logistic regression model stood out because it offered the best overall combination of catching the disease and avoiding false alarms.

The study also looked closely at how these models make their final yes-or-no decisions. By default, many computer programs decide a patient is sick if their calculated risk score is above 50 percent. The researchers tested whether changing this cutoff point to a slightly higher number would improve the results. They found that raising the threshold did not make the model more accurate overall; it simply shifted the balance. It made the model slightly better at avoiding false alarms but slightly worse at catching actual cases. This suggests that for this specific dataset, the standard setting was already quite well-tuned, and simply tweaking the numbers did not solve the underlying limitations of the data.

Beyond raw accuracy, the researchers examined which factors the models relied on most to make their predictions. By systematically shuffling the data to see which pieces were most critical, they discovered that the number of major blood vessels visible in an angiogram, the specific type of chest pain a patient reported, and the results of a thallium stress test were the most powerful predictors. These findings align with established medical knowledge, confirming that the computer models were learning from the most clinically relevant signals rather than random noise. The study also explored how the models performed across different groups, noting that the system worked slightly better for women than for men and showed different patterns across age groups, though the small number of patients in some categories meant these observations were preliminary.

A crucial part of the research involved testing how the models would behave if they were trained on more data. The researchers simulated this by gradually increasing the amount of information the models received during training. They observed that as the training set grew, the models became more consistent and reliable, suggesting that the current performance was limited by the small size of the available data rather than a flaw in the algorithms themselves. The study concludes that while logistic regression is the strongest tool among those tested for this specific dataset, the results cannot be immediately applied to all populations. The data used came from a specific historical collection and does not represent the diverse patient populations found in sub-Saharan Africa or other regions. Before these tools can be used to guide clinical decisions for new groups of people, they must be tested on larger, independent datasets that reflect the real-world diversity of patients. Until then, these findings serve as a rigorous benchmark, showing that careful evaluation and transparency are just as important as the accuracy numbers themselves.

Technical Summary: Comparative Evaluation of Machine Learning Models for Heart Disease Prediction

Problem Statement
Cardiovascular disease remains the leading global cause of death, with a disproportionate burden in low- and middle-income countries. While machine learning (ML) offers potential for early risk identification, studies utilizing the standard UCI Cleveland Heart Disease dataset report widely varying performance metrics. These discrepancies often stem from differences in preprocessing, outcome coding, feature selection, data splitting, and model tuning. Furthermore, many studies rely solely on accuracy, neglecting class-specific performance, calibration, and generalizability. This study addresses the need for a rigorous, multi-metric evaluation of supervised classifiers on the Cleveland dataset to establish a reliable benchmark before attempting generalization to underrepresented populations, such as those in sub-Saharan Africa.

Methodology
The study utilized the UCI Cleveland Heart Disease dataset, comprising 303 records with 13 predictors and an angiographic outcome. The outcome was binary-coded: "disease present" (139 records) for angiographic scores 1–4, and "disease absent" (164 records) for a score of 0.

Data Partitioning: A stratified 70/30 split (random seed 42) created a training set (212 records) and a held-out test set (91 records), preserving class proportions.
Preprocessing: All preprocessing steps (imputation, scaling, categorical encoding) were fitted exclusively on the training data to prevent data leakage. Missing values in the ca (number of major vessels) and thal (thallium test) variables were handled within the training fold.
Algorithms: Five supervised classifiers were implemented using scikit-learn:
1. Logistic Regression (LR) with L2 regularization.
2. Random Forest (RF) with 500 trees and balanced class weights.
3. Gaussian Naive Bayes (GNB).
4. Decision Tree (DT) with balanced class weights.
5. K-Nearest Neighbors (KNN) with $k=5$ .
Evaluation Metrics: Models were ranked primarily by test F1 score and secondarily by ROC-AUC. Additional metrics included accuracy, precision, sensitivity, specificity, Brier score (for calibration), and permutation importance for explainability.
Threshold Analysis: The conventional threshold of 0.50 was compared against a training-derived threshold (0.552) optimized via Youden's J statistic.
Learning Curves: Five-fold cross-validation was used to assess how F1 scores evolved as training sample sizes increased from 33 to 169 records.

Key Results

Model Performance: Logistic Regression (LR) achieved the best overall performance on the test set, with an accuracy of 85.71%, F1 score of 84.34%, ROC-AUC of 0.939, and the lowest Brier score (0.102), indicating superior probabilistic calibration.
- Random Forest (RF) showed comparable discrimination (ROC-AUC 0.933) but lower precision and specificity.
- KNN matched LR's specificity (87.76%) but had lower sensitivity (76.19%).
- GNB achieved the highest sensitivity (90.48%) but poor specificity (51.02%), resulting in the lowest accuracy (69.23%).
- Decision Trees performed the worst (ROC-AUC 0.692).
Threshold Optimization: Raising the classification threshold from 0.50 to 0.552 increased specificity (89.80%) but reduced sensitivity (80.95%) and the F1 score (83.95%). Accuracy remained unchanged at 85.71%. The authors conclude that the 0.552 threshold is an analytical comparison point rather than a clinically superior decision rule.
Explainability: Permutation importance identified the ca (number of major vessels), cp (chest-pain type), and thal (thallium test result) as the top three predictors, with ca causing the largest mean decrease in ROC-AUC (0.067) when shuffled.
Learning Behavior: As the training set size increased, the cross-validation F1 score improved (rising from 0.725 to 0.799), and the gap between training and validation performance narrowed, suggesting that sample size, rather than model complexity, was a limiting factor.
Subgroup Analysis: Exploratory analysis showed higher accuracy for females (93.55%) compared to males (81.67%). The >65 age group showed 100% sensitivity but low specificity (60%), though the authors note these figures are descriptive due to the small sample size (n=9).

Significance and Claims
The paper claims that Logistic Regression serves as the preferred benchmark model for this specific dataset configuration, offering the best balance of discrimination, calibration, and class-specific performance. The study emphasizes that:

Multi-metric evaluation is essential: Relying on accuracy alone obscures critical trade-offs between sensitivity and specificity, as seen in the GNB and DT results.
Calibration matters: LR's low Brier score indicates it provides more reliable probability estimates than non-linear models in this context.
Data constraints limit generalization: The learning curve analysis suggests that performance is constrained by the small sample size (303 records).
Contextual limitations: The authors explicitly state that the findings cannot be generalized to sub-Saharan African populations without external validation using region-specific data, as the Cleveland dataset does not represent local patient mixes or diagnostic pathways.

The study concludes that while LR is robust for this benchmark, future research must prioritize independent validation on larger, diverse datasets before clinical deployment.

Technical Summary: Comparative Evaluation of Machine Learning Models for Heart Disease Prediction

More like this