📄 medicine

Implementation Variability Makes Sepsis-3 an Inconsistent Measuring Model for Early AI Prediction on Clinical Data

This paper demonstrates that implementation variability in Sepsis-3 definitions, particularly regarding suspected infection, significantly undermines the reproducibility and predictive performance of AI models compared to expert-validated labels, thereby necessitating transparent reporting of measurement protocols to ensure comparability in clinical research.

Original authors: Jan Mackensen

Published 2026-08-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jan Mackensen

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the high-stakes environment of an intensive care unit, time is often the difference between life and death. Sepsis, a dangerous reaction to an infection that can cause organ failure, is a leading cause of death in hospitals worldwide. Because the condition can deteriorate rapidly, doctors rely on artificial intelligence to scan patient records and sound the alarm before a patient crashes. To teach these computer systems what to look for, researchers must first teach them what sepsis actually is. They do this by applying a set of agreed-upon medical rules, known as Sepsis-3, to the raw data in electronic health records. These rules act as a measuring stick, turning complex streams of vital signs and lab results into a simple label: sepsis or no sepsis. The hope has been that if everyone uses the same rules, the resulting computer models will be reliable and comparable. However, a new study suggests that this measuring stick is far less consistent than previously thought, raising serious questions about how we train the artificial intelligence that is meant to save lives.

A researcher at Heidelberg University decided to investigate the reliability of these rules by treating the definition of sepsis not as a fixed fact, but as a measuring model. They looked at how different research teams, all claiming to follow the same Sepsis-3 guidelines, actually applied them to patient data. They found that the guidelines leave room for significant interpretation. For instance, the rules say to look for a "suspected infection," but they do not specify exactly what combination of antibiotics and lab tests counts as a suspicion. One team might decide that a single dose of antibiotics is enough, while another might require a full week of treatment. Similarly, the definition of "organ dysfunction" can vary depending on how researchers calculate a specific score called SOFA, which tracks how well a patient's organs are working. The researcher identified six different areas where these choices could be made, and when they combined all the possible variations, they discovered there were over two thousand distinct ways to implement the same single definition.

To see how much these choices mattered, the team ran a massive experiment using data from three different hospital databases. They took the exact same patient records and applied every possible version of the Sepsis-3 rules to them. The results were striking. The different ways of defining a suspected infection were the biggest source of disagreement, followed by how researchers decided exactly when sepsis began. In some cases, the choice of rules changed the final label for a patient completely. The researcher found that these variations were not just minor technical details; they introduced a level of inconsistency that made it difficult to compare the results of different studies. A model trained in one hospital using one set of rules might be fundamentally different from a model trained in another hospital using a slightly different set of rules, even if both claimed to be using the same standard.

The study then took a crucial step by comparing these computer-generated labels against the judgments of real, expert doctors. The researcher used a dataset where senior physicians had manually reviewed patient records and marked exactly when they believed sepsis started. When they compared the various computer versions of Sepsis-3 against these human experts, the computer models struggled to agree with the doctors. The mismatch was particularly severe in the area of sensitivity, meaning the computer rules often failed to identify sepsis cases that the doctors had clearly spotted. This gap in agreement had a direct impact on the performance of the artificial intelligence. Models trained on the computer-generated Sepsis-3 labels performed significantly worse than models trained directly on the expert doctors' labels. In the worst cases, the computer-trained models were up to eighteen percentage points less accurate in their predictions and thirteen points lower in their ability to correctly identify positive cases.

The researcher discovered that they could narrow this performance gap by carefully tuning the computer rules to match the specific way the local experts thought about the disease. By aligning the definition of sepsis onset and infection with the expert labels, they were able to reduce the accuracy gap to just a few percentage points. However, this solution came with a catch. The specific rules that best matched the experts' judgments were often the ones that identified sepsis later in the timeline, which is the opposite of what is needed for early warning systems. This finding suggests that there is no single, perfect version of the Sepsis-3 definition that works for every purpose. A rule set that is excellent for predicting who will die might be terrible for catching the disease early, and a rule set that matches one hospital's experts might fail in another.

Ultimately, the study concludes that the variability in how these rules are applied is a structural feature of modern medicine, not a bug that can be easily fixed. The author argues that the field has relied too heavily on the idea that a consensus definition guarantees a consistent measurement. Instead, they show that the choices made in the code matter just as much as the data itself. For artificial intelligence in healthcare to be trustworthy and reproducible, researchers must stop treating the definition of sepsis as a black box. They need to be transparent about exactly which version of the rules they used and how those rules align with the local clinical reality. Without this clarity, the promise of AI to revolutionize early disease detection remains clouded by the uncertainty of how the disease itself is being measured.

Technical Summary: Implementation Variability Makes Sepsis-3 an Inconsistent Measuring Model for Early AI Prediction on Clinical Data

Problem Statement

The application of machine learning to Electronic Health Records (EHRs) for early disease prediction relies heavily on retrospective labeling using medical consensus definitions. The Sepsis-3 consensus definition is the standard for generating these labels. However, despite relying on the same underlying clinical measurements and consensus text, different research groups produce substantially different sepsis labels due to variations in how the definition is implemented. This paper posits that Sepsis-3 functions as an inconsistent measuring model, where implementation choices introduce significant measurement variability. This variability undermines the comparability of published AI studies and degrades the performance of models trained on these labels when evaluated against site-specific expert ground truth.

Methodology

The study employs a measurement-theoretic perspective, distinguishing between fundamental clinical measurements (vital signs, labs) and derived measurement functions (Suspected Infection, Organ Dysfunction, Sepsis Onset). The research is structured around three experiments:

Literature Analysis and Variant Generation:
- The author analyzed 30 publications to identify implementation differences, categorizing them into six dimensions: SOFA-Version, Data-Imputation, SI-Definition (Suspected Infection), SI-Window, OD-Function (Organ Dysfunction), and Sepsis-Onset.
- A fully crossed design of these dimensions yields 2,520 distinct implementation variants.
Experiment 1: Inter-Definition Comparison:
- Datasets: MIMIC-3, eICU, and SepsisExp.
- Method: Linear Mixed-Effects Models (LMEMs) were used to perform variance component analysis. The implementation components were treated as random effects, and evaluation metrics (Sensitivity, Specificity, MCC, F1, Cosine Similarity) served as response variables.
- Goal: To quantify which components contribute most to the variance in labels when comparing different Sepsis-3 implementations against each other.
Experiment 2: Comparison with Expert Ground Truth:
- Dataset: SepsisExp (containing expert annotations from senior ICU physicians with high inter-rater reliability, $\alpha=0.94$ ).
- Method: Sepsis-3 variants were compared against expert labels using LMEMs to assess agreement (intercepts/BLUPs) and variance contribution.
- Goal: To determine if the variability persists when comparing Sepsis-3 to a clinical reference and to identify which variants align best with expert judgment.
Experiment 3: Impact on AI Prediction Performance:
- Setup: A transformer-based prediction model (adapted from Staniek et al.) was trained to predict sepsis 6 hours in advance.
- Training Data: Nine representative Sepsis-3 implementation variants (selected via clustering of BLUPs) and the expert ground truth.
- Evaluation: Models were evaluated on ground-truth labels. The study measured the performance gap between Sepsis-3-trained models and expert-trained models, analyzing whether better alignment with expert labels (higher Implementation F1) correlated with better prediction performance.

Key Results

1. Sources of Variability (Experiments 1 & 2)

Dominant Factors: The Suspected Infection (SI) Definition is the primary source of variability, accounting for approximately 49–66% of the variance across metrics in MIMIC-3 and SepsisExp. Sepsis-Onset and OD-Function are the next largest contributors.
Minor Factors: SOFA-Version, SI-Window, and Data-Imputation contribute negligible variance (<2% in most cases).
Consistency: The ranking of variability sources remains consistent whether comparing implementations to each other or to expert ground truth.
Specificity vs. Sensitivity: While SI-Definition drives sensitivity variance, the OD-Function becomes the dominant driver of specificity variance.

2. Agreement with Expert Labels (Experiment 2)

Weak Alignment: Sepsis-3 implementations show weak agreement with expert labels, particularly in sensitivity (approx. 20%) and MCC (near 0%).
No Universal Optimal Variant: No single implementation variant is universally optimal.
- For early prediction, variants using min(tSOFA, tSI) (earliest onset) are preferable.
- For phenotyping or mortality analysis, variants using max(tSOFA, tSI) or tSI (later onset) align better with expert labels but are less suitable for early warning.
Trade-offs: Improvements in sensitivity often come at the cost of specificity, with little net gain in MCC.

3. Impact on AI Models (Experiment 3)

Performance Gap: Models trained on Sepsis-3 labels significantly underperform those trained on expert labels. The worst-performing Sepsis-3 variants lagged by up to 18 AUROC points and 13 F1 points.
Alignment Benefit: Aligning the Sepsis-3 implementation to the site-specific expert labels reduced this gap to 3 AUROC points and 2 F1 points.
Statistical Significance: An approximate randomization test showed that models trained on Sepsis-3 labels differed significantly from expert-trained models on almost all test sets, with only the best-aligned variant (Variant 7) showing non-significant differences on a subset of tests.
Data Scarcity: Variance in prediction performance increased when fewer positive sepsis cases were available for training.

Significance and Claims

The paper argues that Sepsis-3 is not a unique, deterministic label generator but an inconsistent measuring model. The author makes the following claims:

Reproducibility Crisis: Published AI results on sepsis are difficult to compare because "identical" Sepsis-3 labels differ substantially across studies due to implementation choices.
Task Dependency: There is no "best" Sepsis-3 implementation. The optimal choice depends on the specific prediction task (e.g., early warning vs. mortality analysis) and the site-specific ground truth.
Methodological Shift: The author advocates against treating discrepancies between Sepsis-3 and expert labels purely as statistical noise to be corrected via domain adaptation or calibration. Instead, they propose a rule-based, interpretable approach: systematically identifying and explicitly modeling the mismatches (e.g., via BLUP analysis) to refine the measurement model for specific patient populations.
Transparency Requirement: To ensure reproducibility and comparability, researchers must transparently report the exact implementation details of consensus definitions and their alignment with site-specific expert labels, alongside open sharing of code and data.

The study concludes that these findings likely extend to other early disease prediction tasks relying on consensus-derived labels, highlighting that variability is a structural feature of medicine rather than an exception.