Development and Validation of An Interpretable Risk Prediction Model for In-Hospital Mortality in Diabetic Patients with Congestive Heart Failure in Intensive Care Unit

Technical Summary: Development and Validation of an Interpretable Risk Prediction Model for In-Hospital Mortality in Diabetic Patients with Congestive Heart Failure in the ICU

Problem Statement
Diabetic patients with congestive heart failure (CHF) represent a high-risk subgroup with elevated in-hospital mortality. Existing prediction models often fail to address this specific comorbidity, typically treating diabetes merely as a confounder rather than modeling the distinct pathophysiology of the DM-CHF interaction. Furthermore, traditional statistical models (logistic/Cox regression) often lack the capacity to capture complex nonlinearities, while many machine learning (ML) approaches suffer from poor interpretability ("black-box" nature), limiting clinical adoption. Additionally, many studies rely on single-center data without rigorous external validation, and few translate risk predictions into actionable, personalized clinical tools. This study addresses the need for a specific, interpretable, and externally validated model to predict in-hospital mortality in this vulnerable population.

Methodology

• Data Sources: The study utilized two large, publicly available critical care databases:
  • Derivation Cohort: MIMIC-IV (Medical Information Mart for Intensive Care IV), containing 7,063 diabetic CHF patients.
  • External Validation Cohort: eICU Collaborative Research Database (eICU-CRD), containing 4,532 diabetic CHF patients.
• Study Population: Patients were identified using ICD-9 code 428.0 for CHF. Inclusion criteria required a diagnosis of diabetes. Exclusion criteria included age <18 and missing outcome data. For patients with multiple admissions, only the first admission was retained.
• Feature Engineering & Selection:
  • Preprocessing: Missing values were imputed using random forest; categorical variables were one-hot encoded; outliers were mitigated via binning.
  • Selection Strategy: A rigorous four-method intersection approach was employed to identify robust predictors. Variables were screened using: (1) LASSO regression, (2) Boruta algorithm, (3) Recursive Feature Elimination (RFE), and (4) Univariate logistic regression. Only variables identified by all four methods were retained.
  • Final Predictors: 13 independent predictors were selected: APS III, age, norepinephrine, vasopressin, anion gap, temperature, respiratory rate, intubation status, SOFA score, mean corpuscular volume (MCV), phenylephrine, and dopamine.
• Model Development:
  • Ten ML algorithms were benchmarked: Gradient Boosting Machine (gbm), LightGBM, Naive Bayes, Ranger, SVM, K-Nearest Neighbors, Logistic Regression, Neural Network, RPART, and XGBoost.
  • Optimal Model Selection: Logistic Regression was selected as the optimal model due to its superior balance of performance and interpretability.
  • Hyperparameter Tuning: A hybrid approach combining Bayesian optimization and manual refinement was used.
  • Validation: The dataset was split into a development set (67%) and a test set (33%). Performance was evaluated using stratified 5-fold cross-validation and an independent test set. External validation was performed on the eICU-CRD cohort.
• Interpretability: The Kernel SHAP (SHapley Additive exPlanations) method was applied to provide both global (feature importance) and local (individual patient) explanations.
• Deployment: An interactive R Shiny web application was developed to facilitate real-time risk prediction and visualization.

Key Results

• Model Performance:
  • Development/Test Set: The optimized logistic model achieved an ROC-AUC of 0.8838 (development) and 0.8663 (independent test set). It demonstrated high specificity (0.9762 in development, 0.9785 in test) and a Brier score of 0.0736 (development) and 0.073 (test), indicating excellent calibration.
  • External Validation: In the eICU-CRD cohort, the model maintained robust performance with an ROC-AUC of 0.7509 and a Brier score of 0.096.
  • Clinical Utility: Decision Curve Analysis (DCA) confirmed that the model provided a net clinical benefit over "treat all" and "treat none" strategies across various threshold probabilities.
• Feature Importance: SHAP analysis identified APS III as the most influential predictor, followed by age, norepinephrine, vasopressin, anion gap, temperature, and respiratory rate.
• Interpretability: The model successfully quantified individual risk contributions. For example, in high-risk cases, elevated APS III and SOFA scores drove the prediction, while protective factors (e.g., absence of vasopressors) lowered risk in low-risk cases.
• Tool Implementation: A web-based tool (https://xaiumedicine.shinyapps.io/DiabeticCHFmortModel/) was deployed, allowing clinicians to input the 13 variables and receive immediate mortality probability estimates accompanied by SHAP-based visual explanations (waterfall and force plots).

Significance and Claims
The authors claim that this study bridges the gap between complex algorithmic prediction and bedside clinical decision-making by:

1. Specificity: Providing a model tailored specifically to the comorbid DM-CHF population, rather than general HF cohorts.
2. Interpretability: Moving beyond "black-box" ML by utilizing logistic regression enhanced with SHAP, ensuring clinicians can understand the "why" behind a prediction.
3. Validation: Demonstrating generalizability through rigorous external validation on a distinct, multi-center dataset (eICU-CRD).
4. Actionability: Translating the model into a user-friendly, mobile-accessible web application that supports real-time risk stratification and personalized intervention planning.

The paper concludes that the developed logistic model is a transparent, accurate, and clinically actionable tool for predicting in-hospital mortality in diabetic patients with CHF, offering a practical solution for early identification of high-risk individuals in the ICU setting.

Limitations Acknowledged by Authors
The authors note that the retrospective nature of the data may introduce bias, residual confounding cannot be fully excluded, and the use of older data (2008–2019) may not reflect current treatment patterns. Additionally, the model focuses solely on in-hospital mortality without long-term follow-up, and certain confounders (e.g., smoking, alcohol use) were unavailable in the public databases.