Variability in Automated Sepsis Case Detection: A Systematic Analysis of Implementation Methods in Clinical Data Repositories

This systematic review reveals that significant methodological heterogeneity in automated sepsis case detection across MIMIC-III and eICU-CRD databases leads to widely varying detection rates, underscoring the urgent need for standardized reporting and open-source code to ensure reproducibility in sepsis research.

The Gist

Imagine you are trying to count how many people in a massive city have a specific type of cold. You have a rulebook that says: "If someone has a fever and their breathing is struggling, they have the cold."

You give this rulebook to 64 different teams of detectives. You expect them to all find roughly the same number of sick people, right?

Wrong.

In this study, the teams found anywhere from 3% to 65% of the population having the cold. That's a huge difference! Some teams found almost everyone was sick; others found almost no one was.

This paper investigates why this happened. The researchers looked at how scientists use computer programs to find "sepsis" (a life-threatening reaction to infection) in hospital records. They discovered that the problem isn't the rulebook (the medical definition); the problem is how the detectives interpret the rulebook while writing their computer code.

Here is the breakdown using some simple analogies:

1. The "Recipe" Problem

Think of the medical definition of sepsis as a recipe for a cake.

• The Rule: "Mix flour, eggs, and sugar. Bake until golden."
• The Reality: The recipe doesn't say how much flour, what kind of eggs, or exactly when to take the cake out of the oven.

In the study, every research team wrote their own version of the recipe.

• Team A used a cup of flour. Team B used a cup and a half.
• Team A checked the oven every 10 minutes. Team B checked every hour.
• Team A assumed if an egg was missing, it was fine. Team B assumed if an egg was missing, the cake was ruined.

Because they followed slightly different steps, they ended up with completely different cakes (different groups of patients), even though they all claimed to be following the same "Sepsis-3" recipe.

2. The "Missing Puzzle Pieces"

Hospital data is messy. It's like a giant puzzle where some pieces are missing, some are upside down, and some are labeled "Unknown."

• The Missing Data: Sometimes a patient's blood pressure wasn't recorded for an hour.
• The Detective's Choice: What do you do?
  • Option A: Assume the patient was fine (give them a "zero" score).
  • Option B: Guess what the number might have been based on the previous hour.
  • Option C: Throw the patient's data out entirely.

The study found that researchers made these guesses differently. One team's "guess" might make a patient look healthy, while another team's "guess" makes the same patient look very sick.

3. The "Time Travel" Confusion

Sepsis happens over time. To catch it, you have to look at a patient's history.

• Team A looked at the last 24 hours before the patient got sick.
• Team B looked at the last 48 hours.
• Team C looked at the moment the patient walked into the ICU.

It's like trying to catch a thief. If you look at the security camera for only 1 minute, you might miss them. If you look for 10 minutes, you might catch them. The researchers were looking at different time windows, so they caught different "thieves" (sepsis cases).

4. The "Copy-Paste" Effect

The researchers also found something interesting: Teams were copying each other.
Some research groups didn't write their own code from scratch; they downloaded code from another group. If the first group made a mistake (or a weird choice), the second group copied it, and the third group copied that too. It's like a game of "Telephone" where the message gets distorted, but in this case, the distortion was baked into the code and spread across the scientific community.

Why Does This Matter?

If you are a doctor trying to build a computer program to warn you about sepsis, you need to know exactly how that program was trained.

• If Program A was trained on a group where 60% of people were sick, and Program B was trained on a group where only 10% were sick, you cannot compare them.
• It's like comparing a basketball player who practiced on a muddy field to one who practiced on a polished court. You don't know who is actually better; you just know they played on different surfaces.

The Big Takeaway

The authors aren't saying the science is bad. They are saying the instructions are too vague.

They are calling for a "Standardized Recipe."

• Before: "Mix the ingredients." (Too vague!)
• After: "Use exactly 200g of flour, check the oven at 350°F, and if a piece of data is missing, fill it in with the average of the last 3 readings."

The Conclusion: To make sure we are all studying the same disease and not just different versions of it, scientists need to stop guessing and start sharing their exact code and step-by-step instructions. Only then can we trust the results.

Technical Summary

1. Problem Statement

Despite the standardization of the Sepsis-3 definition (suspected infection + acute organ dysfunction defined by a $\ge$2 point increase in SOFA score), automated sepsis detection rates vary drastically across studies using identical datasets.

• The Core Issue: When researchers apply the Sepsis-3 definition to the same public intensive care unit (ICU) databases (e.g., MIMIC-III, eICU-CRD), reported sepsis detection rates range from 3.4% to 65.2%.
• The Gap: Previous reviews noted this variability but lacked a systematic characterization of the specific computational implementation decisions (e.g., how missing data is handled, how time windows are defined) that drive these discrepancies.
• Consequence: This heterogeneity undermines the reproducibility of sepsis research, complicates the comparison of machine learning models, and hinders the clinical translation of prediction tools.

2. Methodology

The authors conducted a PRISMA-guided systematic review combined with a source code analysis to investigate methodological heterogeneity.

• Data Sources:
  • Databases: MIMIC-III (46,520 patients) and eICU-CRD (139,367 patients). MIMIC-IV and AmsterdamUMCdb were excluded due to data overlap or insufficient publication volume.
  • Literature Search: PubMed and Web of Science (2016–2024).
  • Selection Criteria: English, open-access, using MIMIC-III/eICU-CRD, explicitly using SOFA-based Sepsis-3 detection, and reporting traceable detection rates.
• Study Cohort:
  • Screened: 396 publications.
  • Included: 64 studies (44 MIMIC-III, 20 eICU-CRD).
  • Source Code: 12 studies provided accessible code; after deduplication and dependency tracing, 6 unique repositories were analyzed in depth.
• Analytical Framework:
  The authors decomposed sepsis detection into six methodological domains (D1–D6):
  1. D1: Parameter Coverage: Selection of clinical parameters and handling of dual information systems (CareVue vs. MetaVision).
  2. D2: Temporal Windows: Sliding window sizes, offsets, and reference points (ICU admission vs. infection onset).
  3. D3: Aggregation Methods: How multiple measurements within a window are summarized (e.g., worst-case vs. mean).
  4. D4: Missing Data Handling: Imputation strategies (e.g., zero-imputation, forward-fill, interpolation).
  5. D5: SOFA Calculation: Baseline assumptions (static 0 vs. dynamic lowest observed) and component modifications.
  6. D6: Infection Detection: Methods to identify suspected infection (ICD codes vs. antibiotic-culture temporal matching).

3. Key Results

A. Extreme Variability in Detection Rates

• MIMIC-III: Detection rates ranged from 3.4% to 65.2% (Median: 26.4%).
• eICU-CRD: Detection rates ranged from 9.8% to 47.9% (Median: 18.7%).
• Within-Population Variability: Even among studies using the exact same source population (e.g., the full MIMIC-III patient table without age filters), rates varied from 16.9% to 42.2%. This proves that patient selection is not the primary driver; rather, implementation logic is.
• No Convergence: There was no trend toward methodological convergence over time (2016–2024), indicating that awareness of the problem has not yet led to standardization.

B. Documentation Gaps

Publications frequently omit critical implementation details, making exact reproduction impossible:

• SOFA Calculation: Only 53.1% of studies reported details.
• Infection Detection: Only 42.2% reported details.
• Temporal Windows: Only 37.5% reported details.
• Aggregation Methods: Only 26.6% reported details.
• Missing Data Handling: Only 17.2% reported details.
• Note: eICU-CRD studies generally had higher documentation rates than MIMIC-III studies, yet variability persisted.

C. Source Code Analysis Findings

Analysis of 6 repositories revealed 321 specific implementation decisions that were often undocumented in the associated papers:

• Baseline SOFA: Some studies assumed a baseline of 0 (interpreting Sepsis-3 as absolute SOFA $\ge$ 2), while others used a dynamic baseline (lowest observed SOFA). This fundamentally changes who is labeled as septic.
• Missing Data: Strategies varied from zero-imputation (assuming missing = normal function, which underestimates severity) to complex multi-stage imputation (linear interpolation, k-NN).
• Infection Detection: Some used antibiotic-culture matching (Seymour et al. method), while others relied solely on ICD-9 codes or APACHE diagnosis criteria.
• Dependency Propagation: Code repositories often build upon one another (e.g., mimic-code $\to$ sepsis3-mimic), propagating specific implementation choices (and their biases) across multiple independent research groups without explicit acknowledgment.

D. Clustering of Results

In eICU-CRD, distinct clusters of studies reported identical detection rates (e.g., 34.94% and 16.60%) across different institutions. This suggests that professional data analytics services or shared codebases may be propagating specific, unverified implementation decisions across the research community.

4. Key Contributions

1. Quantification of Heterogeneity: First systematic quantification of how implementation choices (D1–D6) create massive variability in sepsis cohorts, even with identical definitions and data.
2. Source Code Forensics: Moving beyond text analysis to inspect actual code, revealing that >65% of methodological decisions are undocumented in publications.
3. Identification of Propagation: Demonstrated that methodological choices are often inherited through code dependencies, creating "echo chambers" of specific implementation biases.
4. Framework for Standardization: Proposed a structured six-domain framework (D1–D6) to categorize and report sepsis detection methodologies.

5. Significance and Recommendations

• Reproducibility Crisis: The study highlights that "standardized definitions" do not guarantee standardized results in computational epidemiology. Prediction models trained on different "sepsis" labels may appear to perform similarly but are actually predicting different clinical phenotypes.
• Impact on Future Definitions: The upcoming SOFA-2 definition will likely exacerbate this heterogeneity if not accompanied by strict standardization.
• Recommendations:
  1. Mandatory Reporting: Authors must report details for all six domains (D1–D6) in supplementary materials.
  2. Version-Controlled Code: Publication of source code is essential for reproducibility.
  3. Reference Implementations: Development of community-agreed, reference implementations for major databases (MIMIC, eICU) to serve as a baseline.
  4. Guideline Integration: Update reporting guidelines (e.g., TRIPOD, STROBE) to include specific checklists for sepsis case detection methodology.

Conclusion: The variability in sepsis detection is not due to stochastic noise but to divergent, often undocumented, computational choices. To improve the robustness of sepsis research and clinical decision support systems, the field must move from "conceptual standardization" to "implementation standardization."