Patient-specific antibiotic susceptibility ranking for empirical treatment of gram-negative bloodstream infection: a retrospective clinical decision-support and transportability study

Technical Summary: Patient-Specific Antibiotic Susceptibility Ranking for Empirical Treatment of Gram-Negative Bloodstream Infection

Problem Statement
Empirical antibiotic selection for hospital-acquired gram-negative bloodstream infections (HA-GN-BSI) presents a high-stakes clinical challenge. Clinicians must initiate active therapy before definitive susceptibility results are available, balancing the risk of delayed treatment against the ecological pressure of indiscriminate broad-spectrum use. While cumulative antibiograms provide population-level susceptibility data, they fail to capture individual patient risk profiles, such as prior antibiotic exposure, recent microbiology results, or specific admission trajectories. Although machine learning has been proposed to generate "personalized antibiograms," existing evidence often lacks rigorous evaluation of recommendation-level performance, comparisons with observed clinician prescribing, and external transportability across different healthcare systems.

Methodology
The authors conducted a retrospective clinical decision-support modeling study using two distinct datasets:

1. Development and Internal Validation: The MIMIC-IV v3.1 dataset was used to develop and evaluate models for adult HA-GN-BSI episodes. The cohort consisted of 1,004 episodes from 962 unique patients, split at the subject level into training (603), validation (201), and test (200) sets.
2. External Transportability: The Antibiotic Resistance Microbiology Dataset Mass General Brigham (ARMD-MGB) v1.0.0 was used to assess the transportability of drug-specific prediction signals.

Key Design Features:

• Decision Point: Analysis was anchored to the index blood-culture collection. Predictors were restricted to information available before this time (demographics, encounter context, prior microbiology, prior antibiotic exposure, and pre-index labs/vitals).
• Target Drugs: The primary recommendation set included piperacillin-tazobactam, cefepime, and meropenem. Ciprofloxacin and gentamicin were modeled as secondary targets.
• Modeling Approach: Separate drug-specific models were developed using logistic regression and gradient-boosted trees (GBT).
• Ranking Strategy: Three ranking rules were compared on the validation set: all-logistic-regression, all-GBT, and a per-drug AUROC-selected strategy. The all-GBT rule was selected for the final test set based on superior Top-1 coverage.
• Comparators:
  • Meropenem-for-all: A high-coverage baseline strategy.
  • Observed Clinician Therapy: Retrospective medication administration records from 6 hours before to 24 hours after culture collection served as a prescribing-context benchmark (not a gold standard for optimal therapy).
• Evaluation Metrics: Performance was assessed via AUROC, AUPRC, Brier score, and calibration metrics. Recommendation-level outcomes focused on Top-1 and Top-2 active coverage (proportion of episodes where the recommended drug was susceptible). Decision Curve Analysis (DCA) was performed per drug.

Key Results

• Internal Performance (MIMIC-IV):

  • In the complete-panel test set (192 episodes), the selected all-GBT ranking rule achieved a Top-1 active coverage of 91.1% and a Top-2 coverage of 96.4%.
  • The "Meropenem-for-all" strategy achieved 93.2% coverage. The model did not miss any episodes covered by the "Meropenem-for-all" strategy but missed four that the latter covered.
  • Meropenem Dominance: The model's high coverage was driven almost entirely by meropenem selection. Meropenem accounted for 90.1% of Top-1 recommendations. Only 7.8% of episodes received a non-meropenem Top-1 recommendation that was also active.
  • Comparison with Clinicians: The model demonstrated significantly higher apparent active coverage than observed clinician therapy (91.1% vs. 39.6%; risk difference +51.6%, p < 0.001). However, this came with a massive disparity in meropenem use: 90.1% for the model vs. 9.9% for clinicians. Clinicians achieved more non-meropenem active therapy (34.9%) compared to the model (7.8%).
  • Calibration: Calibration was imperfect across models (slopes < 1.0), suggesting the framework is better suited as a ranking tool than a precise probability calculator.

• External Transportability (ARMD-MGB):

  • Drug-specific discrimination persisted in the external dataset, particularly for cefepime (GBT AUROC 0.754).
  • However, calibration remained imperfect, and the study noted that the external dataset could not replicate the exact bedside episode definition or variable set of MIMIC-IV. Thus, the results support drug-specific transportability rather than the direct transfer of the full prescribing policy.

• Decision Curve Analysis:

  • DCA indicated that for meropenem, high baseline susceptibility made "treat-all" strategies appear favorable across many thresholds, limiting the incremental net benefit of the model. For cefepime and secondary targets, the model offered clearer value over "treat-all" at higher thresholds.

Key Contributions

1. Holistic Evaluation Framework: The study moves beyond simple discrimination metrics (AUROC) to evaluate the policy of antibiotic selection. It explicitly reports the trade-off between active coverage and carbapenem exposure.
2. Empirical vs. Model Comparison: By benchmarking against observed clinician prescribing within the same candidate drug space, the study highlights the gap between algorithmic "optimal" coverage (driven by broad-spectrum agents) and real-world stewardship practices.
3. Transportability Evidence: The study provides evidence that susceptibility prediction signals can travel across health systems for specific drugs, even if the full ranking policy requires local recalibration.
4. Stewardship Warning: The authors demonstrate that a model can achieve high retrospective active coverage primarily by recommending the broadest-spectrum agent (meropenem), which may not align with antimicrobial stewardship goals.

Significance and Claims
The paper concludes that patient-specific susceptibility ranking can produce high retrospective active coverage, but this performance is largely achieved through meropenem-heavy recommendations. The authors argue that empirical antibiotic decision support should not be evaluated solely on prediction accuracy or active coverage. Instead, such systems must report active coverage, non-meropenem active therapy, and carbapenem exposure together before clinical implementation.

The study asserts that while drug-specific prediction signals are transportable, the prescribing policy built on those signals is not directly transferable without local testing, recalibration, and stewardship controls. The authors caution that without explicit rules to limit broad-spectrum escalation, improved susceptibility prediction may inadvertently increase ecological pressure rather than improve patient outcomes. The work serves as a call for cautious interpretation, emphasizing that better prediction does not automatically equate to better stewardship.