Technical Summary: Machine Learning–Based Identification of Drug-Resistant Pathogen Infection in Patients With Sepsis

Problem Statement

In the intensive care unit (ICU), clinicians face a critical decision-making window at admission: initiating empiric antimicrobial therapy before microbiology culture and susceptibility results are available. This is particularly challenging when multidrug-resistant organisms (MDROs)—specifically methicillin-resistant Staphylococcus aureus (MRSA), vancomycin-resistant Enterococcus (VRE), and carbapenem-resistant Pseudomonas aeruginosa (CRPA)—are suspected. Inaccurate early risk assessment can lead to delayed active therapy or unnecessary broad-spectrum antibiotic exposure, driving further resistance. While electronic health record (EHR) data offers a potential resource for early risk stratification, existing machine learning (ML) models often suffer from methodological flaws, including unclear outcome timing, data leakage from post-index variables (e.g., length of stay), incomplete microbiology definitions, and a lack of interpretability or assessment of clinical utility.

Methodology

The authors conducted a retrospective multicenter cohort study utilizing two large, publicly available ICU databases: eICU Collaborative Research Database (v2.0) and MIMIC-IV (v3.1).

Data Cohort and Definitions

• Population: Adult ICU admissions (age ≥18) with an ICU length of stay ≥24 hours.
• Exclusions: Patients with known MDRO infections or positive MDRO cultures prior to ICU admission were excluded to ensure the models predict new infections arising during the stay.
• Outcome Window: The primary outcomes were binary indicators of culture-positive infection for MRSA, VRE, and CRPA. Outcomes were restricted to clinically ordered cultures obtained from ICU admission through ICU day 7. Surveillance or screening cultures were excluded to minimize misclassification of colonization as infection.
• Predictors: A total of 168 candidate predictors were extracted from the first 24 hours of ICU data. These included demographics, comorbidities (Elixhauser), early organ-support therapies, vital signs, laboratory tests, and severity scores (e.g., SOFA). Crucially, post-index variables (such as ICU or hospital length of stay) were strictly excluded to prevent data leakage and ensure clinical actionability at admission.

Model Development and Validation

• Data Split: The merged cohort (N=45,141) was randomly split into a development set (70%, n=31,599) and an independent validation set (30%, n=13,542).
• Feature Selection: The Boruta algorithm was applied to the training set to identify robust features by comparing them against permuted shadow features, ensuring only variables with predictive signal were retained.
• Algorithms: Five classifiers were trained for each pathogen: Penalized Logistic Regression (LASSO), AdaBoost, LightGBM, XGBoost, and CatBoost. Hyperparameters were tuned via resampling within the training set.
• Handling Imbalance: Given the low prevalence of resistant infections (1.5% for MRSA, 0.7% for VRE, 0.2% for CRPA), the study employed class-weighting and threshold optimization strategies. Missing data were imputed using median (continuous) or mode (categorical) values derived strictly from the training set.
• Interpretability: SHapley Additive exPlanations (SHAP) were used to quantify feature contributions to individual predictions and summarize global importance.
• Evaluation Metrics: Performance was assessed using the Area Under the Receiver Operating Characteristic Curve (AUROC), sensitivity, specificity, and Positive Predictive Value (PPV). Decision Curve Analysis (DCA) was employed to evaluate clinical utility across threshold probabilities (0–50%), comparing model-guided strategies against "treat-all" and "treat-none" approaches.

Key Results

Prevalence and Baseline Characteristics

In the prespecified early ICU window, the prevalence of culture-positive infections was:

• MRSA: 1.5% (n=683)
• VRE: 0.7% (n=296)
• CRPA: 0.2% (n=90)

Unadjusted comparisons revealed significant differences in demographics, laboratory values, and physiologic status between infected and non-infected patients, providing the basis for predictive modeling.

Model Performance

All model families demonstrated clinically meaningful discrimination on the independent validation set:

• MRSA: The LASSO model achieved the highest AUROC of 0.83 (95% CI 0.77–0.87), with sensitivity and specificity both at 0.75. Gradient-boosted methods (LightGBM, XGBoost, CatBoost) performed similarly (AUROC 0.81–0.82).
• VRE: AdaBoost and CatBoost achieved the highest AUROCs of 0.85 (95% CI 0.82–0.89). CatBoost showed a specificity of 0.84 and sensitivity of 0.74.
• CRPA: LightGBM provided the best discrimination with an AUROC of 0.82 (95% CI 0.77–0.86), followed by XGBoost (0.81).

Interpretability (SHAP)

SHAP analysis identified pathogen-specific predictive signals, highlighting that different biological mechanisms drive risk for each organism:

• MRSA: Top contributors included eosinophil count, PaO2, platelet count, white blood cell count, hemoglobin, and glucose.
• VRE: Key drivers were fibrinogen, lymphocyte count, PaCO2, white blood cell count, and platelet count.
• CRPA: Significant features included potassium, LDL cholesterol, lymphocyte count, ALT, oxygenation gradients, and pH.

Clinical Utility

Decision Curve Analysis demonstrated that all three models yielded a positive net benefit across clinically relevant threshold probabilities (0–50%) compared to treat-all or treat-none strategies.

• For MRSA, LASSO showed the greatest net benefit in the 10–35% threshold range.
• For VRE, CatBoost showed consistent net benefit across most thresholds.
• For CRPA, LightGBM dominated at moderate-to-high thresholds (~20–50%).

Significance and Claims

The paper claims that explainable machine-learning models based on routinely available early ICU variables can effectively stratify the risk of documented early MRSA, VRE, and CRPA infections in sepsis patients.

The authors emphasize that their approach addresses specific limitations in prior literature by:

1. Strictly defining the outcome window (ICU days 0–7) to match the clinical decision timeline.
2. Excluding post-index variables (like length of stay) to prevent data leakage and ensure the model is actionable at admission.
3. Providing transparency through SHAP values, offering patient-level explanations rather than "black box" predictions.
4. Assessing clinical utility via Decision Curve Analysis, moving beyond simple discrimination metrics to evaluate potential net benefit in clinical workflows.

The authors conclude that while these models show promise for supporting targeted diagnostics, infection-control precautions, and empiric therapy decisions, external and prospective validation is required before clinical implementation. They explicitly state that these models are intended to augment, not replace, clinician judgment and microbiology testing.