First-24-hour machine learning for 30-day mortality prediction in ICU trauma patients: development in MIMIC-III and cross-database evaluation in MIMIC-IV

Technical Summary: First-24-Hour Machine Learning for 30-Day Mortality Prediction in ICU Trauma Patients

Problem Statement
ICU trauma patients exhibit significant clinical heterogeneity, making early risk stratification for mortality challenging. While conventional severity scores capture important burden components, they often fail to represent the nonlinear interactions and short-term physiologic trajectories characterizing early deterioration. Existing machine learning (ML) studies in this domain frequently suffer from methodological limitations, including the use of variables unavailable at the time of prediction (temporal leakage), reliance on internal validation only, and an overemphasis on accuracy or AUROC rather than metrics better suited for imbalanced outcomes. Furthermore, cross-database validation remains uncommon, limiting evidence regarding model transportability across different clinical eras and database structures. This study addresses the need for a robust, early-warning framework that utilizes only information available within the first 24 hours of ICU admission to predict 30-day mortality, while rigorously evaluating performance across different versions of the MIMIC database.

Methodology
The study employed a retrospective cohort design using the MIMIC-III and MIMIC-IV databases.

• Cohort Definition: Adult patients (18–89 years) with trauma-related diagnosis codes were included. The prediction window was strictly the first 24 hours post-ICU admission. Patients who died, recovered, or were discharged before 24 hours were excluded to ensure a complete observation window. The target outcome was death within 30 days of ICU admission.
• Feature Construction: Predictors were restricted to the first 24 hours to prevent temporal leakage. Variables included demographics, neurologic, respiratory, hemodynamic, renal, metabolic, hematologic, and treatment-intensity indicators. For repeatedly measured variables, temporal statistics (mean, min, max, standard deviation, first/last values, change, slope) and measurement counts were computed. Clinically motivated ratios (e.g., BUN-to-creatinine) and interactions were also included.
• Model Development (MIMIC-III):
  • An 80:20 patient-level split (3,411 training, 853 hold-out) was used.
  • Six candidate feature configurations were trained using XGBoost.
  • The "all-expanded temporal" configuration was selected based on the highest Area Under the Precision–Recall Curve (AUPRC) in the hold-out subset.
  • Hyperparameters included 1,100 estimators, max depth of 5, and specific regularization settings. No class weighting was applied in this internal selection phase.
  • The operating threshold (0.3282) was selected by maximizing the F1 score using out-of-fold predictions.
• Cross-Database Evaluation (MIMIC-IV):
  • A harmonized feature space of 228 predictors (common to both MIMIC-III and IV) was established.
  • A new XGBoost model was refitted on the complete MIMIC-III cohort (4,264 patients) using these 228 features, incorporating class weighting to address imbalance.
  • This model was evaluated on 13,747 ICU stays in MIMIC-IV without using MIMIC-IV outcomes for any model development, hyperparameter tuning, or recalibration.
• Evaluation Metrics: Primary metric was AUPRC (due to class imbalance). Secondary metrics included AUROC, Brier score, calibration plots, and SHAP (SHapley Additive exPlanations) for interpretability.

Key Results

• Internal Performance (MIMIC-III Hold-out): The selected model achieved an AUPRC of 0.5563 (95% CI: 0.4720–0.6436) and an AUROC of 0.8633 (95% CI: 0.8306–0.8941). The Brier score was 0.0857. At the selected threshold, sensitivity was 51.3% and specificity was 92.0%.
• Cross-Database Performance (MIMIC-IV): The harmonized model evaluated in MIMIC-IV achieved an AUPRC of 0.4952 (95% CI: 0.473–0.519) and an AUROC of 0.8249 (95% CI: 0.816–0.834). The Brier score was 0.109.
• Calibration: In MIMIC-IV, calibration was monotonic but showed increasing overprediction at higher risk levels. In the highest predicted-risk decile, the mean predicted probability was 0.7352 versus an observed mortality of 0.5651.
• Interpretability (SHAP): Age was the most influential predictor, followed by respiratory-rate measurement frequency, maximum Glasgow Coma Scale (GCS) score, 24-hour urine output, and respiratory-rate variability. The model identified clinically plausible associations, such as lower GCS, lower urine output, and elevated lactate correlating with higher mortality risk.

Significance and Claims
The paper claims that clinical information recorded during the first 24 hours after ICU admission retains predictive value for 30-day mortality across different versions of the MIMIC database (MIMIC-III and MIMIC-IV). The study demonstrates that a harmonized ML framework can maintain discriminatory power (AUROC > 0.82) when transported to a new database version without retraining on the target data.

However, the authors explicitly frame these findings as evidence from a research-stage prediction framework rather than a deployment-ready clinical model. They emphasize several limitations that restrict generalizability:

1. Optimism in Internal Estimates: The internal performance metrics may be optimistic because the same hold-out subset was used for both feature configuration selection and performance reporting.
2. Non-Identical Models: The performance drop in MIMIC-IV cannot be attributed solely to dataset shift, as the cross-database model differed in feature space (228 vs. 259 predictors), training sample size, and class weighting.
3. Calibration Issues: The model overpredicts risk at higher levels in MIMIC-IV, and raw probabilities are not calibrated for the target population.
4. Documentation and Reproducibility: Incomplete feature-mapping documentation and differences in cohort construction (e.g., handling of repeated hospitalizations) limit reproducibility.
5. Single-Center Provenance: Both databases originate from the same medical center, limiting external validity to other institutions.

The authors conclude that while first-24-hour data is informative, fully independent model selection, reproducible feature harmonization, target-population recalibration, and multicenter validation are required before clinical implementation.