ICU Mortality and LOS Prediction Models Using Machine Learning Based on Both Real and Synthetic Data

Technical Summary: ICU Mortality and LOS Prediction Models Using Machine Learning Based on Both Real and Synthetic Data

Problem Statement
The development of accurate machine learning (ML) models for Intensive Care Unit (ICU) outcomes in low-resource settings, specifically in Ethiopia, is hindered by two primary challenges: a scarcity of high-quality training data and significant class imbalance. In the studied cohort of 10,669 patients from five Ethiopian public hospitals, the mortality rate was only 13.9%, creating a severe imbalance that typically biases models toward the majority class (survival). Furthermore, standard severity indices (e.g., SAPS-II, SOFA) are often inapplicable in this context due to the unavailability of specific laboratory tests (e.g., arterial blood gas, bilirubin) and resource constraints affecting timely interventions. While deep learning architectures (e.g., Transformers, TCNs) excel in high-resource environments with dense time-series data, they are limited in settings where data is sparse, tabular, and often paper-based. This study addresses the need for predictive tools that function effectively within these data-constrained environments.

Methodology
The study employed a retrospective comparative experimental design using data from five public hospitals in Ethiopia (2013–2016 Ethiopian Calendar). The dataset included 10,669 adult ICU records covering demographics, vital signs, laboratory results, and outcomes.

1. Data Preprocessing and Balancing:

   • Records with >20% missingness in key administrative fields were dropped.
   • Missing numerical values were imputed with the median; categorical values with the mode.
   • To address class imbalance, the majority class (survivors) was undersampled, creating a balanced training set of 2,942 patients (50% mortality). This process discarded 7,631 original records.
   • Outliers were capped at the 1st and 99th percentiles, and categorical features were label-encoded.

2. Synthetic Data Generation:
   Three distinct algorithms were evaluated for generating synthetic data to augment the real dataset:

   • SMOTE-NC (Synthetic Minority Over-sampling Technique for Nominal and Continuous): Used linear interpolation between existing minority instances and their k-nearest neighbors. It is computationally efficient (2.1s) and deterministic.
   • CTGAN (Conditional Tabular Generative Adversarial Network): A generator-discriminator architecture designed for complex, non-linear tabular distributions. It is computationally intensive (187.4s) and stochastic.
   • Variational Autoencoder (VAE): A probabilistic latent variable model. It struggled with mixed data types in this specific implementation, resulting in poor performance.

3. Experimental Design:
   Models were trained under three configurations:

   • Real-Only (RO): Training exclusively on real data.
   • Synthetic-Only (SO): Training exclusively on synthetic data.
   • Hybrid (HY): Training on a combination of 80% real and 20% synthetic data.

4. Machine Learning Algorithms:
   Three algorithms were tested for both classification (Mortality) and regression (Length of Stay - LOS):

   • Logistic Regression (Baseline)
   • Random Forest
   • XGBoost

5. Validation:
   A stratified 80/20 hold-out split was used, with the test set (n=589) sequestered until final evaluation. Performance was assessed using Accuracy, F1-score, AUC for mortality, and Mean Absolute Error (MAE) and R² for LOS. Stability was verified via 5×5 repeated stratified cross-validation.

Key Results

• Synthetic Data Quality:

  • CTGAN demonstrated superior downstream utility, achieving 91.7% accuracy on a real test set (vs. 86.4% for SMOTE-NC), though it required significantly longer training time.
  • VAE failed to produce predictive power (51.6% accuracy, comparable to random guessing) and was excluded from further hybrid experiments.
  • SMOTE-NC offered a pragmatic balance of high correlation preservation and computational efficiency.

• Mortality Prediction:

  • Hybrid models consistently outperformed real-only and synthetic-only models.
  • The best performing model was CTGAN + XGBoost in the hybrid configuration, achieving an accuracy of 0.998 (95% CI: 0.995–1.000), F1-score of 0.996, and AUC of 0.99.
  • SMOTE-NC + Random Forest also achieved high accuracy (0.996).
  • Note: The authors explicitly caution that these high accuracy figures are based on an artificially balanced test set (50% mortality) and do not reflect real-world class distributions.

• Length of Stay (LOS) Prediction:

  • CTGAN + XGBoost in the hybrid configuration achieved the best performance with an MAE of 4.08 days (95% CI: 3.58–4.67) and an R² of 0.601.
  • CTGAN + Random Forest trained on synthetic-only data surprisingly outperformed real-only baselines (MAE 3.76 days), suggesting CTGAN successfully preserved clinically meaningful temporal patterns.

• Feature Importance:

  • Oxygenation emerged as the dominant predictor. Oxygen requirement (r = 0.327) and SpO2 (r = 0.299) were the top-ranked features.
  • Hemoglobin showed negligible association with mortality (r = -0.023, rank 15/19).
  • Age was not a statistically significant predictor (p = 0.39).
  • SHAP analysis confirmed that oxygenation variables drive mortality risk more than hemoglobin or age in this specific cohort.

Significance and Claims
The paper claims to make three primary contributions:

1. Comparative Performance: It provides a systematic comparison of machine learning models across real, synthetic, and hybrid data configurations in a low-resource Ethiopian setting.
2. Context-Specific Predictors: It identifies clinical predictors specific to the Ethiopian ICU context, notably highlighting oxygenation as the critical factor while challenging the significance of hemoglobin and age found in other studies.
3. Prototype Development: It developed an interactive Streamlit prototype to visualize model insights, feature importance, and "what-if" scenarios for healthcare stakeholders.

The authors emphasize that hybrid modeling (combining real and synthetic data) is a viable solution for creating effective prediction models in resource-limited settings where data is scarce and imbalanced. They argue that synthetic data augmentation can improve model discrimination without introducing significant bias, provided the generation method (e.g., CTGAN or SMOTE-NC) is validated for downstream utility.

Modesty and Limitations
The paper maintains a modest tone regarding clinical readiness:

• Validation Status: The results are based on internal validation (retrospective data from five hospitals). The authors explicitly state that the high accuracy (99.5%) is likely optimistic due to the artificial balancing of the test set and the lack of external validation on unseen institutions.
• Clinical Utility: The prototype is described as a "proof-of-concept" for research demonstration and hypothesis generation only. It is not approved for clinical use and should not inform patient care decisions.
• Generalizability: The findings are specific to public tertiary hospitals in Ethiopia and should not be generalized to private hospitals, rural facilities, or other countries without recalibration and external validation.
• Causality: The authors clarify that SHAP values and feature importance indicate association, not causation. The observed relationships (e.g., between SpO2 and mortality) are confounded by unmeasured factors like time to intervention and resource availability.

The study concludes that while synthetic data augmentation shows promise for overcoming data scarcity, prospective external validation and implementation science trials are prerequisites before any clinical deployment.