💻 computer science

Prevalence Shift and ICU Mortality Miscalibration Across Institutions

This paper demonstrates that hospital-level prevalence mismatch is a dominant, quantifiable cause of calibration failure in cross-institutional ICU mortality models, showing that a simple Bayesian correction using only local mortality rates can effectively restore calibration without requiring labeled target data.

Original authors: Aman Chandra H, Stuti Shivhare, Suyash Sahu

Published 2026-07-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Aman Chandra H, Stuti Shivhare, Suyash Sahu

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a weather forecaster who has spent years perfecting your predictions in Boston. You know exactly how often it rains there. Your model is excellent: when you say there's a 70% chance of rain, it actually rains 70% of the time. You are "calibrated."

Now, imagine you pack up your weather model and move it to Arizona. In Arizona, it rains very rarely. But your model, trained on Boston's history, still thinks, "Well, it's cloudy, so there's a 70% chance of rain!"

In Arizona, you will be wrong almost every time. You will predict rain constantly, but the sky will stay clear. Your model is still good at ranking days (it knows which days are cloudier than others), but it has completely lost its ability to tell you the actual probability of rain.

This paper is about exactly that problem, but instead of weather, it's about predicting patient survival in Intensive Care Units (ICUs).

The Core Problem: The "Boston vs. Arizona" Effect

The researchers took a machine learning model trained on data from one hospital (MIMIC-IV, which had a patient death rate of about 11%) and tested it across 185 different hospitals in the US.

They found that these hospitals were very different. Some had death rates as low as 0%, while others were as high as 20%.

When the model was used in a hospital where the death rate was much lower than 11%, it started "over-predicting" death. It told doctors, "This patient has a 20% chance of dying," when in reality, the chance was only 5%. This is dangerous because it could lead to unnecessary, aggressive treatments.

The Big Surprise: The "Speedometer" vs. The "Thermometer"

Usually, when scientists check if a model is working, they look at a metric called AUROC. Think of AUROC as a speedometer. It tells you if the model is good at ranking patients (e.g., "Is Patient A sicker than Patient B?").

The paper found that the speedometer (AUROC) was fine. The model was still great at ranking patients from "sick" to "less sick," even in the new hospitals.

However, the thermometer (called Calibration or ECE) was broken. The thermometer was giving the wrong temperature. The model was saying "It's 100 degrees" when it was actually "50 degrees."

The Key Takeaway: Just because a model is good at ranking patients (Speedometer) doesn't mean it gives the right probabilities (Thermometer). In fact, the paper shows that the more the death rate in the new hospital differed from the training hospital, the more broken the thermometer became.

The Solution: A Simple "Mathematical Adjustment"

The researchers discovered that the model wasn't broken because it was "stupid"; it was broken because it was holding onto the wrong "prior belief" (the 11% death rate from Boston).

They found a simple, free fix. It's like a mathematical translator.

If you know the actual death rate in your local hospital (let's say it's 5%), you can plug that single number into a simple formula. This formula instantly "re-tunes" the model's predictions.

Before: The model says "20% chance of death."
After the fix: The model says "5% chance of death."

Why is this amazing?

It requires zero new data: You don't need to feed the model thousands of new patient records. You just need the hospital's current death rate (a number hospitals already track).
It works instantly: You don't need to retrain the model or hire a data scientist.
It's as good as expensive methods: The paper showed this simple fix worked almost as well as methods that require 30% of the hospital's patients to be manually labeled and checked by experts.

The "Why" and the "How Much"

To prove this wasn't just a coincidence, the researchers created a "fake" scenario where they only changed the death rate in a computer simulation, keeping everything else exactly the same. The model's predictions got worse only because the death rate changed. This proved that the difference in death rates was the sole cause of the broken predictions.

They also calculated that for well-designed models, about 60% of the prediction errors were caused simply by this difference in death rates. The other 40% was due to other differences between hospitals (like different equipment or patient types), which are harder to fix.

Summary

The Issue: AI models trained in one hospital often give wrong probability estimates when used in another hospital because the "death rate" is different.
The Trap: Standard tests (AUROC) say the model is fine because it still ranks patients correctly, hiding the fact that the probabilities are wrong.
The Fix: A simple, free math formula that uses the local hospital's death rate to instantly correct the model's predictions.
The Result: This "zero-label" fix makes the model accurate again without needing new data or retraining, saving hospitals from making dangerous over-estimations of patient risk.

Technical Summary: Prevalence Shift and ICU Mortality Miscalibration Across Institutions

Problem Statement
Machine learning models for predicting in-hospital mortality in Intensive Care Units (ICUs) often demonstrate strong discriminative performance (measured by AUROC) on their training data but suffer from performance degradation when deployed across different hospital systems. While "dataset shift" is a known challenge, the specific impact of prevalence shift (also termed label shift or prior probability shift) on model calibration remains under-characterized at scale.

Prevalence shift occurs when the marginal distribution of the outcome, $P(Y)$ , differs between the training source and the deployment target, while the conditional distribution of features given the outcome, $P(X|Y)$ , remains stable. In clinical settings, miscalibration is critical: overestimation leads to unnecessary interventions, while underestimation delays essential care. The paper posits that hospital-level variations in mortality rates act as a primary mechanistic driver of calibration failure, yet this relationship has not been systematically quantified across a large, diverse set of real-world institutions. Furthermore, standard metrics like AUROC are theoretically invariant to monotonic probability transformations and thus insensitive to prevalence shifts, creating a "blind spot" where models appear robust (high AUROC) but are clinically unreliable (poor calibration).

Methodology
The study utilizes a cross-institutional deployment framework involving two major datasets:

Source Domain: MIMIC-IV (Beth Israel Deaconess Medical Center), containing 45,587 patients for training.
Target Domain: eICU Collaborative Research Database, spanning 185 distinct US hospitals with 159,224 patients.

Experimental Design:

Models: Four classifiers representing different families were trained on MIMIC-IV: Logistic Regression (LR), Random Forest (RF), XGBoost (XGB), and LightGBM (LGBM).
Evaluation: Models were evaluated across the 185 eICU hospitals. Performance was measured using the Area Under the Receiver Operating Characteristic Curve (AUROC) and the Expected Calibration Error (ECE).
Correlation Analysis: The study analyzed the correlation between the "prevalence gap" ( $|\pi_h - \pi_0|$ , where $\pi_h$ is the local hospital mortality rate and $\pi_0$ is the training prevalence) and both ECE and AUROC.
Correction Strategy: A closed-form Bayesian prevalence correction was applied. This method adjusts the predicted probabilities using only the local mortality rate ( $\pi_h$ ) and the training prevalence ( $\pi_0$ ), requiring no labeled target data or model retraining. The formula is derived from the label-shift framework:
$\hat{p}_h(x) = \frac{\hat{p}(x) \cdot (\pi_h/\pi_0)}{\hat{p}(x) \cdot (\pi_h/\pi_0) + (1-\hat{p}(x)) \cdot \frac{1-\pi_h}{1-\pi_0}}$
Comparative Baselines: The zero-label Bayesian correction was compared against:
- No correction (raw predictions).
- Supervised methods (Platt scaling, Isotonic regression, Temperature scaling) fitted on 30% of labeled target data per hospital.
Causal Validation: Synthetic experiments were conducted on held-out MIMIC-IV data where prevalence was artificially shifted via stratified resampling while strictly preserving $P(X|Y)$ . This isolated prevalence shift as the sole variable to confirm causality.
Attribution Analysis: An empirical metric ( $\phi_h$ ) was defined to quantify the fraction of total calibration error attributable specifically to prevalence shift.

Key Results

Strong Correlation between Prevalence Gap and Calibration Error: Across all four model architectures, ECE showed a strong positive correlation with the prevalence gap ( $r = 0.52$ to $0.75$, $p < 10^{-14}$ ). As the difference between local and training mortality rates increased, calibration error increased significantly.
AUROC/ECE Dissociation: In sharp contrast, AUROC showed weak to non-significant correlations with the prevalence gap ( $|r| = 0.03$ to $0.22$). For LightGBM, the correlation was effectively zero ( $r = -0.027$ ), confirming that rank-based metrics fail to detect calibration degradation caused by prevalence shift.
Effectiveness of Zero-Label Correction: The Bayesian prevalence correction reduced mean ECE by 56% to 65% across well-calibrated models (LR, XGB, LGBM).
- Example: For LightGBM, ECE dropped from 0.085 to 0.037.
- This performance was comparable to supervised methods (Platt scaling and Isotonic regression) that utilized 30% labeled target data (ECE $\approx$ 0.022–0.024).
- Random Forest showed lower improvement (43.7% reduction) due to inherent overconfidence unrelated to prevalence shift.
Causal Confirmation: Synthetic experiments confirmed that varying prevalence alone (while holding $P(X|Y)$ constant) was sufficient to reproduce the linear increase in ECE observed in real-world deployments.
Attribution Decomposition: Empirical analysis revealed that prevalence shift accounts for approximately 55–65% of the total calibration error for well-calibrated models. The remaining error is attributed to other factors like feature drift or measurement heterogeneity.

Significance and Claims
The paper does not claim novelty in the mathematical derivation of the Bayesian correction formula, which is grounded in established label-shift theory (Saerens et al., Lipton et al.). Instead, the novel contributions are empirical and quantitative:

Large-Scale Quantification: This is the first study to systematically characterize the relationship between hospital-level outcome prevalence and calibration error across 185 real institutions and 159,224 patients.
Demonstration of the Blind Spot: It provides robust evidence that AUROC is insufficient for assessing deployment readiness in the presence of prevalence shift, as models can maintain high discrimination while becoming severely miscalibrated.
Practical, Zero-Label Solution: The work establishes a practical framework for clinical deployment where hospitals can correct miscalibration using a single, readily available population statistic (local mortality rate) without requiring labeled data, retraining, or machine learning expertise.
Causal Attribution: It empirically validates that prevalence shift is the dominant driver of calibration error in this setting, explaining the majority of the degradation observed in cross-institutional deployment.

The authors conclude that prevalence mismatch is an analytically tractable source of deployment failure that can be immediately mitigated by any institution using the proposed zero-label correction, thereby improving the safety and reliability of AI-driven clinical decision support.