Benchmarking Machine Learning Architectures for Antimicrobial Stewardship in Pediatric ICUs

Technical Summary: Benchmarking Machine Learning Architectures for Antimicrobial Stewardship in Pediatric ICUs

Problem Statement
Antimicrobial resistance is a critical global threat, necessitating effective Antimicrobial Stewardship (AMS) programs to reduce unnecessary antibiotic exposure. In Pediatric Intensive Care Units (PICUs), clinical uncertainty often leads to the default initiation of broad-spectrum antibiotics, contributing to overuse and long-term resistance. While Machine Learning (ML) has been proposed to support AMS by predicting intervention opportunities from Electronic Health Record (EHR) data, prior work has largely focused on adult populations and static tabular representations. There is a significant gap in understanding how to effectively model AMS interventions in pediatric settings, where data is more heterogeneous and sparse. Furthermore, it remains unclear which intervention targets are most informative and which modeling approaches (tabular vs. temporal) best capture the clinical structure of stewardship decisions.

Methodology
The authors present a systematic benchmarking study evaluating ML architectures for predicting AMS interventions in PICUs. The study utilizes two distinct cohorts: a public dataset (PIC, from the Children's Hospital of Zhejiang University) and a private institutional dataset.

• Prediction Setting: The task is formulated as a patient-day classification problem. At the start of each day with active antibiotic therapy, the model predicts whether a stewardship intervention will occur within the subsequent 24 hours.
• Targets: Four clinically relevant proxy targets are defined based on retrospective prescription data:
  1. IV-to-oral switching: Transition from intravenous to non-intravenous formulation.
  2. De-escalation: Switching to an antibiotic with a strictly lower spectrum (based on the Antibiotic Spectrum Index).
  3. Discontinuation: Termination of therapy prior to discharge with no immediate restart.
  4. Short-course therapy: Total antibiotic duration between 24 and 96 hours (excluding prophylactic cefazolin), serving as a proxy for potential over-treatment.
• Data Representation: The study compares three temporal resolutions:
  1. 24-hour (Tabular): Summary statistics (min, max, mean, count) of dynamic variables aggregated over the preceding day.
  2. 1-hour (Augmented): Hourly bins of the last 24 hours encoded via a Convolutional Neural Network (CNN) and concatenated with the 24-hour features.
  3. Irregular (Graph-based): Native, irregularly sampled EHR data processed without discretization.
• Model Classes: Three classes of models are evaluated under a unified framework:
  • Tabular Models: Logistic Regression, LightGBM, MLP, and TabPFN.
  • Sequence-based Temporal Models: GRU, LSTM, TCN, Transformer, and State Space Models (SSM/Mamba-inspired). Both Single-Task Learning (STL) and Multi-Task Learning (MTL) configurations are tested.
  • Graph-based Temporal Models: RAINDROP, which models multivariate time series as a graph over variables.
• Evaluation: Models are evaluated using AUROC, F1 Score, and Area Under the Precision-Recall Curve (AUPRC). Calibration is also assessed to determine the reliability of probability estimates.

Key Results

• Target Prevalence Drives Performance: Model performance is primarily determined by target prevalence and data characteristics rather than model complexity. Rare targets (e.g., IV-to-oral switching, occurring in <1% of cases) yield poor precision-recall performance despite high AUROC scores, highlighting the limitations of AUROC in imbalanced clinical settings.
• Temporal Modeling Benefits: Sequence-based models provide improvements in precision-recall trade-offs (AUPRC and F1) over tabular approaches, particularly for more prevalent targets like de-escalation and short-course therapy. However, these gains are modest when moving from 24-hour aggregated data to finer 1-hour temporal resolutions.
• Calibration Trade-off: A significant trade-off exists between discrimination and calibration. While complex sequence and graph-based models often achieve better ranking performance, they are consistently less well-calibrated than simpler tabular models (particularly LightGBM). Tabular models produce more reliable probability estimates, which is critical for clinical decision support.
• Multi-Task Learning (MTL): Jointly modeling related stewardship tasks yields only marginal improvements over single-task learning. This suggests that the shared predictive structure across different intervention types (e.g., de-escalation vs. discontinuation) is weak, and task-specific modeling remains preferable.
• Graph-Based Limitations: The graph-based model (RAINDROP) showed subpar performance compared to sequence models, suggesting that its approach to modeling irregular sampling did not fully capture day-level patterns in this specific context.

Significance and Claims
The paper claims to provide the first systematic, cross-cohort evaluation of AMS intervention prediction specifically for pediatric populations. Its primary significance lies in offering practical guidance for developing reliable Clinical Decision Support Systems (CDSS) for AMS:

1. Target Selection is Critical: Standard adult targets (like IV-to-oral switching) may be underutilized or rare in pediatric settings, necessitating the definition of pediatric-specific proxies like short-course therapy.
2. Model Complexity vs. Utility: Increasing model complexity (e.g., using deep temporal models) does not guarantee superior performance. Simpler tabular models often provide more reliable probability estimates (calibration), which is essential for clinician trust.
3. Temporal Representation: While finer temporal resolution offers incremental gains, the majority of predictive signal is captured by aggregated daily features.
4. Evaluation Metrics: The study emphasizes that AUROC alone is insufficient for evaluating clinical utility; metrics like AUPRC and calibration are more informative for decision support tools.

The authors conclude that for PICU AMS intervention prediction, target definition, temporal representation, and calibration are as critical as model choice, and that increasing model complexity alone is insufficient for building reliable clinical decision support systems. The code for the study is made publicly available to ensure reproducibility.