Observation-process features are associated with larger domain shift in sepsis mortality prediction: a cross-database evaluation using MIMIC-IV and eICU-CRD

This study demonstrates that while incorporating observation-process features like measurement counts improves internal discrimination in sepsis mortality prediction models, it significantly exacerbates domain shift and degrades external calibration and transportability across different ICU databases.

The Gist

Imagine you are a chef who has perfected a recipe for a delicious soup in your own kitchen. You know exactly how the ingredients taste, how long they simmer, and how the heat affects the flavor. Your soup is a hit with your family (this is internal performance).

Now, imagine you try to sell this soup recipe to a restaurant in a completely different city. You expect it to taste the same, but something goes wrong. The soup tastes "off," maybe too salty or not spicy enough, even though you used the exact same recipe. Why? Because the water quality, the type of stove, and even the way the local farmers grow the vegetables are different (this is domain shift).

This paper is about a similar problem, but instead of soup, it's about computer programs (AI models) that predict if a patient with sepsis (a severe infection) will survive or die in the hospital.

The Big Problem

Doctors and data scientists have built many AI models to predict sepsis outcomes. These models work great in the hospital where they were built. But when they are moved to a different hospital, they often fail. They might still guess "who will die" correctly (discrimination), but they get the probability wrong (calibration). They might say a patient has a 90% chance of dying when the real chance is only 30%. This is dangerous because doctors rely on these numbers to make life-or-death decisions.

The "Secret Ingredient" That Causes Trouble

The researchers in this paper wanted to know why these models fail when moved to new hospitals. They suspected the problem wasn't just the patient's biology, but how the data was collected.

Think of a patient's vital signs (heart rate, temperature, blood pressure) as the "ingredients."

• The Ingredient: The actual heart rate (e.g., 100 beats per minute).
• The Observation Process: How often the nurse checks the heart rate.

In Hospital A, nurses might check vitals every hour. In Hospital B, they might check every 4 hours.

• If a patient is very sick, Hospital A's nurses might check them every 15 minutes.
• If a patient is sick in Hospital B, they might still only check every 4 hours.

The AI model learns that "checking vitals very often = very sick." But this isn't a biological truth; it's a habit of the hospital staff. When the model moves to a new hospital with different habits, it gets confused.

What They Did

The researchers built 7 different versions of a sepsis prediction model using data from two massive databases:

1. MIMIC-IV: Data from one big hospital in Boston (The "Home Kitchen").
2. eICU: Data from 208 different hospitals across the US (The "New Cities").

They tested different ways of feeding data to the AI:

• Simple Model: Just the average vital signs.
• Complex Model: The highest and lowest vital signs, how much they fluctuated, and how many times the nurses checked them (the "measurement counts").

The Surprising Findings

Here is the twist:

1. At Home (Boston): The more complex models were better. Adding "how often nurses checked" the patient helped the AI guess the outcome more accurately. It was like adding a secret spice that made the soup taste perfect in the home kitchen.
2. In New Cities (External Validation): The more complex models failed harder.
   • The models that included "how often nurses checked" (measurement counts) crashed the hardest when moved to new hospitals.
   • The models that just looked at the patient's biology (without the "checking habits") were more stable, even if they were slightly less accurate at home.

The Analogy:
It's like teaching a student to take a test.

• Simple Model: Teaches the student the actual math concepts. They might get a B, but they can pass the test in any school.
• Complex Model: Teaches the student the math concepts plus the specific handwriting style of the teacher and the exact time the teacher asks questions. They get an A+ in that specific class. But if they take a test in a different school with a different teacher, they fail miserably because they were memorizing the process, not the truth.

The Takeaway

The paper concludes that feature engineering is a trade-off.

• If you want your model to be the absolute best at the hospital where it was built, you should include "observation-process" features (like how often things were measured).
• BUT, if you want your model to work reliably in other hospitals, you should be very careful about including those features. They act like "local dialects" that confuse the model when it travels.

The Golden Rule: Before deploying a medical AI in a new hospital, don't just ask "Is it accurate?" Ask "Is it calibrated?" (Does it give the right percentage of risk?). The study found that calibration is the first thing to break when you use features that reflect hospital habits rather than pure biology.

In short: Don't let your AI learn the habits of one hospital if you want it to work everywhere else. Stick to the biology.

Technical Summary

1. Problem Statement

Clinical prediction models for sepsis often suffer from domain shift (or dataset shift) when deployed outside their development environment. While internal performance (discrimination) is often high, external validation frequently reveals poor calibration and reduced transportability.

• The Gap: A critical, underappreciated mechanism for this degradation is that Electronic Health Record (EHR) data encodes not only patient physiology but also the observation process (e.g., measurement timing, frequency, and missingness patterns).
• The Hypothesis: These observation-process features are predictive within a specific hospital due to local workflows and documentation habits but are unstable across sites. The authors hypothesize that enriching models with complex physiologic summaries and observation-process features (like measurement counts) improves internal discrimination but exacerbates external domain shift and calibration failure.

2. Methodology

Study Design & Data Sources

• Design: Retrospective cohort study with a controlled experimental framework.
• Derivation Cohort: MIMIC-IV (Single-center, academic tertiary care, Boston, 2008–2022). $N = 30,218$ sepsis patients (16.3% mortality).
• External Validation Cohort: eICU-CRD (Multi-center, 208 US hospitals, 2014–2015). $N = 31,403$ sepsis patients (13.9% mortality).
• Population: Adult ICU patients meeting Sepsis-3 criteria (suspected infection + acute SOFA increase $\ge$ 2) with an ICU stay $\ge$ 24 hours.
• Outcome: In-hospital mortality.

Feature Engineering & Model Specifications

The study compared seven prespecified model specifications to isolate the impact of feature complexity and observation-process proxies. All models used variables aligned with the APACHE III framework.

1. Physiologic Summaries:
   • Simple: APACHE III score only.
   • Intermediate: Most recent values (Latest).
   • Complex: Minimum/Maximum values (Min/Max).
   • Highly Complex: Within-window variability (Max - Min).
2. Observation-Process Features:
   • For each physiologic variable, the measurement count (number of recorded values in the first 24 hours) was added as a feature in paired specifications.
3. Algorithms:
   • Logistic Regression (LR): Used as the primary transparent baseline.
   • Gradient-Boosted Trees (XGBoost): Used to assess non-linear interactions and robustness.

Evaluation Metrics

• Discrimination: Area Under the Receiver Operating Characteristic Curve (AUROC) and Area Under the Precision-Recall Curve (AUPRC).
• Calibration: Calibration slope, intercept, and Brier score.
• Domain Shift: Quantified as $\Delta$AUROC (External AUROC - Internal AUROC). Negative values indicate performance degradation.

3. Key Results

Internal Performance (MIMIC-IV)

• Trend: Internal discrimination improved as feature complexity increased.
• Impact of Counts: Adding measurement counts consistently boosted internal AUROC.
  • Example (LR): Model 2 (Latest values) AUROC = 0.819 $\rightarrow$ Model 3 (Latest + Counts) AUROC = 0.834.
  • Example (XGBoost): Model 2 AUROC = 0.849 $\rightarrow$ Model 3 AUROC = 0.859.

External Performance & Domain Shift (eICU-CRD)

• The Trade-off: Models with higher internal performance suffered larger drops in external performance.
• Observation-Process Impact: Including measurement counts significantly increased the magnitude of domain shift ($\Delta$AUROC).
  • Logistic Regression:
    • Model 2 (No counts): $\Delta$AUROC = -0.047.
    • Model 3 (With counts): $\Delta$AUROC = -0.082.
    • Model 7 (Variability + Counts): $\Delta$AUROC = -0.125 (Worst performance).
  • XGBoost: Showed smaller incremental degradation from counts compared to LR but still exhibited significant domain shift in complex specifications.

Calibration Degradation

• Calibration Slope: Decreased progressively with model complexity.
  • Logistic Regression: Slope dropped from 1.007 (Model 1, APACHE only) to 0.417 (Model 7, Variability + Counts). A slope < 1 indicates overfitting to the derivation set and over-prediction of risk in the external set.
• Sensitivity: Calibration metrics were more sensitive to domain shift than discrimination metrics. The inclusion of measurement counts caused the most severe miscalibration.

Algorithm Comparison

• XGBoost vs. Logistic Regression: XGBoost demonstrated greater robustness to observation-process shifts in "Latest" and "Min/Max" specifications. This is likely because tree-based models use binary splits (thresholding) which can handle distributional shifts in counts better than the linear coefficients of Logistic Regression. However, XGBoost still failed significantly when using "Variability" features.

4. Key Contributions

1. Quantification of Observation-Process Risk: The study provides empirical evidence that features encoding how data is collected (measurement frequency) are a primary driver of domain shift, distinct from the physiological signal itself.
2. Feature Engineering Trade-off: It establishes a clear trade-off: richer feature sets (complex summaries + counts) yield better internal discrimination but worse external generalizability and calibration.
3. Calibration as a Sentinel: The authors demonstrate that calibration slope is the most sensitive indicator of reduced transportability, deteriorating before discrimination metrics show significant failure.
4. Algorithmic Nuance: The study highlights that while tree-based models (XGBoost) are more robust to certain types of observation-process shifts than linear models, they are not immune, particularly when features encode unstable monitoring patterns (variability).

5. Significance & Implications

• For Model Developers: Feature selection for models intended for multi-site deployment must critically evaluate whether a feature represents a stable biological signal or a site-specific workflow artifact. Adding "measurement counts" or "missingness" features may improve local performance but render the model unusable elsewhere.
• For Clinical Deployment: External validation must prioritize calibration assessment alongside discrimination. A model with high AUROC but poor calibration (slope < 0.8) is dangerous for clinical decision-making.
• Future Directions: The findings suggest that domain adaptation strategies should specifically target observation-process features. Furthermore, model developers should consider using simpler physiologic summaries or robust algorithms (like XGBoost with careful feature selection) when deploying across diverse healthcare systems.

Conclusion: Enriching sepsis prediction models with observation-process features creates a "local optimization" trap. While these features capture valuable workflow-specific signals that boost internal metrics, they introduce instability that severely compromises the model's ability to generalize, making calibration the critical metric for assessing real-world readiness.