Development and external validation of a laboratory-free clinical decision-support model for sepsis prediction at emergency department triage: a retrospective cohort study

Technical Summary: Development and External Validation of a Laboratory-Free Sepsis Prediction Model

Problem Statement
Early identification of sepsis in the emergency department (ED) is critical for reducing mortality, yet remains challenging due to the heterogeneous presentation of the condition. Existing bedside tools, such as the quick Sequential Organ Failure Assessment (qSOFA), demonstrate suboptimal sensitivity in unselected ED populations, potentially missing a significant proportion of cases during triage. While machine learning (ML) models have been developed for sepsis prediction, many rely on laboratory values, microbiology results, or complex temporal features that are unavailable at the moment of patient arrival. This limits their utility for triage nurses who must make disposition decisions within minutes. There is a need for a "laboratory-free" clinical decision-support model that relies exclusively on nurse-assessable variables, requires no specialist interpretation, and possesses the potential to generalize across different institutions.

Methodology
This retrospective cohort study adhered to the TRIPOD guidelines and utilized two large, publicly available critical care databases accessed via PhysioNet:

• Development Cohort: MIMIC-IV-ED (version 2.2), comprising 139,393 ED encounters from Beth Israel Deaconess Medical Center (2011–2019).
• External Validation Cohort: eICU-CRD (version 2.0), restricted to 1,128 patients admitted to ICUs from EDs across 208 US hospitals (2014–2015).

Outcome Definition: Sepsis was defined using ICD-9-CM and ICD-10-CM diagnosis codes consistent with the Sepsis-3 empirical approach (septicemia, severe sepsis, septic shock) appearing in the discharge diagnosis of the index hospitalization.

Predictor Variables: The study focused exclusively on variables available to triage nurses within the first 5–10 minutes of arrival, excluding all laboratory, imaging, and microbiology data. The feature set included:

1. Demographics (age, sex).
2. Triage vital signs (temperature, heart rate, respiratory rate, SpO2, systolic/diastolic BP, pain score).
3. Emergency Severity Index (ESI) acuity level.
4. Chief complaint category (categorized via keyword matching).
5. Longitudinal vital sign trends (min, max, mean values during the ED stay).

Model Development:

• Algorithms: Four classifiers were evaluated: Logistic Regression (L2 regularized), Random Forest, Gradient Boosting, and XGBoost.
• Preprocessing: Median imputation for continuous variables, one-hot encoding for categorical variables, and z-score standardization. Class imbalance was addressed via class-weighting or scale-pos-weight adjustment.
• Validation Strategies:
  • Internal Validation: Stratified random split (70% training, 30% testing).
  • Temporal Validation: Chronological split (earliest 70% training, most recent 30% testing) to assess robustness against secular trends.
  • External Validation: Application of the trained models to the eICU-CRD cohort without modification of coefficients or thresholds.
• Simplified Model: A parsimonious five-variable model (age, systolic BP, heart rate, respiratory rate, chief complaint) was developed to maximize feasibility for low-resource settings.
• Benchmarks: Performance was compared against qSOFA, National Early Warning Score (NEWS), and Modified Early Warning Score (MEWS).

Key Results

• Internal Performance: XGBoost achieved the highest discrimination in the MIMIC-IV-ED test set with an AUROC of 0.875 (95% CI 0.871–0.879), significantly outperforming qSOFA (0.614), NEWS (0.719), and MEWS (0.697). At the optimal threshold (0.416), the XGBoost model achieved 81.4% sensitivity and 98.5% negative predictive value (NPV), compared to qSOFA's 5.7% sensitivity.
• Temporal Stability: The XGBoost full model maintained an AUROC of 0.875 in the temporal test set, confirming that performance was not driven by random data splits or transient institutional patterns.
• Simplified Model Performance: The five-variable simplified model retained strong discrimination (AUROC 0.800) in internal and temporal validation, representing a modest but significant decrement from the full 18-variable model.
• External Validation: In the eICU-CRD cohort, the full model's performance dropped significantly (AUROC 0.555), performing comparably to qSOFA (0.588; p=0.72). Calibration was poor in external validation (slope 0.16). However, the simplified five-variable model achieved a higher AUROC (0.648) than both the full model and qSOFA in this external setting, suggesting superior transportability.
• Feature Importance: Triage systolic blood pressure, heart rate, respiratory rate, age, and chief complaint were the dominant predictors.

Significance and Claims
The authors claim that this study successfully developed and validated a laboratory-free sepsis prediction model that significantly outperforms established early warning scores (qSOFA, NEWS, MEWS) in internal and temporal settings. The study highlights that:

1. High Sensitivity: The ML model offers substantially higher sensitivity than qSOFA while maintaining a high negative predictive value, which is crucial for triage workflows aimed at ruling out low-risk patients.
2. Transportability of Simplicity: Contrary to the expectation that complex models generalize better, the simplified five-variable model demonstrated superior cross-institutional transportability in external validation. The authors attribute this to reduced susceptibility to feature distribution shifts when fewer variables are transported across divergent data architectures.
3. Clinical Feasibility: The model relies entirely on standard triage data, making it suitable for integration into electronic health records, basic smartphone applications, or printed decision aids without requiring additional hardware or blood draws.
4. Limitations and Future Needs: The authors modestly acknowledge that while the model shows promise, external validation performance was imperfect, likely due to spectrum bias (ICU-admitted cohort vs. general ED), feature proxy mismatches (using ICU early vitals as a proxy for triage vitals), and institutional heterogeneity. They conclude that prospective validation and local calibration are required before clinical deployment, and that the tool should be viewed as a complement to, rather than a replacement for, existing clinical judgment and infrastructure.