Comparative Analysis of Machine Learning Models vs. Traditional Clinical Calculators for Cardiovascular Risk Prediction

Technical Summary: Comparative Analysis of Machine Learning Models vs. Traditional Clinical Calculators for Cardiovascular Risk Prediction

Problem Statement
Cardiovascular diseases (CVD) remain the leading cause of global mortality, yet primary prevention strategies rely heavily on traditional risk calculators (e.g., Framingham, ASCVD, SCORE, HEARTS). A critical limitation of these tools is their derivation from high-income European and North American populations, leading to poor calibration and predictive accuracy when applied to diverse epidemiological contexts, particularly Hispanic/Latino communities. While machine learning (ML) offers the potential to capture non-linear interactions in biomedical data, its application in clinical CVD prediction faces challenges regarding severe class imbalance (survivors vastly outnumbering events) and the need for rigorous probability calibration to ensure clinical utility. This study addresses the gap in comparative performance between state-of-the-art ML models and eleven conventional clinical calculators within a diverse US cohort.

Methodology
The study utilized data from the National Health and Nutrition Examination Survey (NHANES) 1999–2018, linked with the National Death Index for mortality follow-up through 2019.

• Dataset: A final analytical cohort of 12,847 participants was derived from approximately 230,000 raw records after a nine-stage cleaning pipeline. The outcome was cardiovascular mortality (15.9% event rate).
• Variables: The model utilized 16 predictor variables (including age, sex, blood pressure, lipids, diabetes, smoking) and 5 engineered features (e.g., BMI, Atherogenic Index), totaling 21 inputs for tree-based models.
• Model Development: Six algorithms were evaluated: Logistic Regression, Cox Proportional Hazards, Gradient Boosting, AdaBoost, Random Forest, and Extra Trees. These were combined with six class-balancing strategies (including random oversampling at 1:1, 2:1, 3:1, undersampling, and SMOTE), yielding 36 distinct model configurations.
• Calibration: To address the artificial inflation of event prevalence caused by oversampling, predicted probabilities were corrected using the Saerens prior probability adjustment method.
• Comparison: The ML models were benchmarked against eleven conventional calculators (ASCVD Original/Colombia, Framingham Original/Colombia, High/Low SCORE, SCORE2, HEARTS Caribbean/North America/Colombia, PROCAM, and REGICOR).
• Evaluation Metrics: Performance was assessed using AUC-ROC, sensitivity, specificity, F1-score, and a weighted composite score (40% AUC, 20% sensitivity, 20% F1, 10% Kappa, 10% inverse Brier). Statistical significance of AUC differences was determined using DeLong's test.
• Tooling: A web-based platform, "CardioPrediQ," was developed to integrate these calculators and ML models, operating entirely client-side to ensure data privacy.

Key Results

• Model Performance: The Gradient Boosting Machine (GBM) with 2:1 oversampling and Saerens calibration achieved the highest performance (AUC = 0.8934; composite score = 0.7904). The top six positions in the composite ranking were exclusively occupied by ML and statistical models.
• Comparison with Traditional Calculators: The best-performing traditional calculator was ASCVD Original (AUC = 0.8843). DeLong's tests revealed that the GBM model significantly outperformed calculators derived from non-US populations (High/Low SCORE, Framingham, PROCAM, REGICOR; p < 0.05). However, the difference between the GBM model and the top-performing traditional calculators (ASCVD Original, HEARTS Caribbean, ASCVD Colombia, SCORE2, HEARTS North America) was not statistically significant.
• Trade-offs: Traditional calculators like ASCVD Original and HEARTS Caribbean exhibited high sensitivity (0.89–0.95) but lower specificity (0.59–0.71), indicating a tendency toward risk overprediction. In contrast, the GBM model maintained a more balanced trade-off (Sensitivity: 0.8515; Specificity: 0.8167).
• Calibration: The Saerens calibration method substantially improved model calibration, reducing the Brier score from 0.1286 to 0.1158.
• Feature Importance: Age was the dominant predictor (41.2% relative importance), followed by systolic blood pressure (18.7%) and total cholesterol (9.4%). Engineered features contributed modestly, suggesting primary clinical variables suffice for most discriminative capacity.

Significance and Claims
The paper claims that ensemble ML models with appropriate class-balancing and probability calibration achieve modest but statistically significant improvements in cardiovascular risk discrimination compared to traditional calculators, particularly when the latter are derived from populations with different epidemiological profiles.

• Context-Dependent Superiority: The study validates that while ML models outperform European-derived calculators in this US cohort, their advantage over US-derived calculators (like ASCVD) is statistically negligible in terms of discrimination alone.
• Calibration Utility: A primary contribution is the demonstration that Saerens calibration effectively corrects the systematic risk overestimation often seen in ML models trained on oversampled data, transforming them into tools with excellent probability calibration suitable for shared decision-making.
• Tool Integration: The "CardioPrediQ" platform is presented as a scalable solution for integrated risk assessment, allowing clinicians to cross-validate estimates across multiple tools and ML models without data leaving the local device.
• Implications for Latin America: The findings support the continued use of region-specific tools like WHO HEARTS in Latin American settings while suggesting that calibrated ML models could offer a viable alternative where locally derived outcome data for recalibration are unavailable.

The authors conclude that while ML offers superior composite performance, its integration into clinical practice must be accompanied by rigorous calibration to ensure accurate risk estimation and avoid overtreatment. The study emphasizes that the choice of risk calculator should be guided by the derivation population of the target cohort.