📄 medicine

An EHR Data Passport for Renal-Risk AI: Cross-Database Transportability of Creatinine-Based AKI Prediction in Intensive Care

This study introduces an EHR data passport framework to evaluate the cross-database transportability of creatinine-based AKI prediction models between MIMIC-IV and eICU, revealing that while discrimination remains stable, significant differences in data computability, laboratory coverage, and calibration shifts necessitate rigorous pre-deployment stress testing and algorithmic vigilance.

Original authors: Xiaoxi Zhang, Shaonan Wang, Giselle Chan, Lixin Yin

Published 2026-07-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiaoxi Zhang, Shaonan Wang, Giselle Chan, Lixin Yin

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart weather forecaster. This forecaster is trained in New York City (let's call this the "MIMIC" database) to predict when a storm (Acute Kidney Injury, or AKI) is going to hit. The forecaster is pretty good at spotting storms in New York.

Now, you want to take this same forecaster and send them to Chicago (the "eICU" database) to predict storms there. You might think, "Great! A storm is a storm, right? If it works in New York, it should work in Chicago."

This paper is essentially a stress test to see if that weather forecaster actually works when moved to a new city. The authors found that while the forecaster can still rank who is most likely to get wet (it knows who is at higher risk than who), it gets the actual amount of rain completely wrong.

Here is the breakdown of their findings using simple analogies:

1. The "Data Passport" Idea

Before you let a machine learning model (the AI) travel from one hospital system to another, the authors say you need to give it a Data Passport.

Think of this like a travel visa. Before you can enter a country, you need to prove you have the right documents, the right vaccinations, and that your background checks out.

The Passport Check: The authors created a checklist to see if the new hospital's data is even ready to be used. They asked: "Do we have enough blood test results? Are the tests done at the same time? Are the numbers missing in the same way?"
The Finding: They found that the "New York" hospital (MIMIC) took blood tests very frequently and recorded them perfectly. The "Chicago" hospital (eICU) took fewer tests and had more missing records. Because the "rules of the road" were different, the AI got confused.

2. The "Storm" (The Prediction)

The AI's job is to predict Acute Kidney Injury (AKI). In the real world, this happens when a patient's kidneys stop working well.

How the AI learns: It looks at a patient's first 24 hours in the ICU. It checks their age, gender, and blood test results (like creatinine, which measures kidney function).
The Problem: In the "New York" hospital, doctors ordered blood tests very often. In the "Chicago" hospital, they ordered them less often.
The Glitch: The AI learned a secret trick in New York: "If a patient gets a lot of blood tests, they are probably sick." When the AI went to Chicago, it saw patients who didn't get as many tests and thought, "Oh, they aren't getting tested, so they must be fine." But in reality, they might be very sick; the doctors just didn't order the test as often.

3. The "Over-Confident" vs. "Under-Confident" Forecaster

When the AI traveled between the two databases, it didn't just get slightly less accurate; it got biased in opposite directions:

New York AI in Chicago: It became over-confident. It started screaming "STORM!" for everyone. It told the Chicago doctors, "This patient has a 40% chance of kidney failure!" when the real chance was only 20%. This leads to alarm fatigue—doctors start ignoring the alerts because they are always crying wolf.
Chicago AI in New York: It became under-confident. It looked at high-risk patients in New York and said, "Nah, they're probably fine," when they were actually in danger. This is dangerous because it might miss patients who need immediate help.

4. Why "Ranking" Isn't Enough

The paper points out a common mistake in AI research. Many people only look at a score called AUROC (which is like asking: "Did the AI correctly guess that Patient A is sicker than Patient B?").

The AI was actually okay at this! It could still tell who was sicker than whom.
But, for a doctor, knowing who is sicker isn't enough. They need to know the exact risk. If the AI says "50% risk" but the real risk is "20%," the doctor might make the wrong decision about how much medicine to give or whether to move the patient to a special unit.

5. The Main Lesson: "Algorithmovigilance"

The authors coin a term called Algorithmovigilance. Think of this as "monitoring the health of the AI."

Just like a doctor monitors a patient's vital signs, hospitals need to monitor their AI.
Before you trust an AI to make decisions in a new hospital, you must check its Data Passport. You need to verify that the way they measure things (how often they draw blood, how they record missing data) matches the way the AI was trained.

Summary

The paper concludes that building a kidney-risk AI is not just about writing a clever computer program. It's about understanding the data environment.

If you take a model trained in a hospital that tests patients constantly and drop it into a hospital that tests patients less often, the model will fail—not because it's "dumb," but because the rules of the game changed.

The takeaway: Don't just ask, "Is this AI accurate?" Ask, "Does this AI's passport match the hospital it's trying to work in?" If the data doesn't line up, the AI might be giving you a false sense of security or causing unnecessary panic.

Problem Statement

The clinical deployment of Artificial Intelligence (AI) models for Acute Kidney Injury (AKI) prediction faces a critical barrier: the assumption that performance metrics (specifically discrimination) derived from one Electronic Health Record (EHR) system will hold true when transported to another. While models may preserve rank ordering (AUROC), they often fail to maintain absolute risk calibration across different institutions due to variations in data provenance, measurement practices, endpoint computability, and missingness patterns.

The paper argues that AKI prediction is not merely a modeling challenge but a data construction problem. The AKI label itself is dependent on observation intensity (how often creatinine is measured), baseline availability, and documentation practices. Consequently, a model trained on public datasets like MIMIC-IV or eICU may learn patterns of "measurement frequency" rather than pure "renal physiology," leading to systematic over- or under-prediction of risk when deployed in a new setting. Current reporting often emphasizes discrimination (AUROC) while obscuring these transportability failures.

Methodology

The study employed a retrospective secondary analysis using two large, public ICU databases: MIMIC-IV v3.1 and eICU-CRD v2.0.

1. Cohort and Endpoint Construction:

Population: Adult ICU stays.
Outcome: A "leakage-controlled" 7-day operational AKI definition. Baseline creatinine was the lowest value in the first 24 hours post-admission. AKI was defined as a post-24h value (up to day 7) reaching $\ge$ 1.5x baseline or increasing by $\ge$ 0.3 mg/dL. This definition intentionally excludes urine output (due to eICU data limitations) to focus on creatinine-based computability.
Predictors: A compact, shared set including age, sex, first-24h renal/electrolyte labs (Creatinine, BUN, Na, K, Cl, HCO3, Anion Gap, Ca), missingness indicators for these labs, and the count of first-24h creatinine measurements.

2. The "AI-Ready Data Passport":
The authors introduced a structured framework called the EHR Data Passport, a set of pre- and post-modeling checks designed to assess data readiness before transport. It evaluates six domains:

Cohort eligibility
Endpoint computability
Shared laboratory coverage
Measurement density
Missingness strata
External calibration behavior

3. Modeling and Validation Strategy:

Algorithm: L2-regularized logistic regression (chosen for transparency over algorithmic novelty).
Validation: A bidirectional cross-database validation was performed.
- Model A: Trained on MIMIC-IV, validated internally (holdout) and externally on eICU.
- Model B: Trained on eICU, validated internally (holdout) and externally on MIMIC-IV.
Handling: Continuous variables were median-imputed and standardized based on the development set. No recalibration was performed during this stage to expose raw transportability failure modes.

4. Evaluation Metrics:
Beyond AUROC and AUPRC, the study prioritized calibration (intercept, slope, decile-level observed vs. predicted risk) and failure-mode analysis (stratifying errors by measurement density and missingness).

Key Results

1. Data Readiness and Computability Disparities:

MIMIC-IV showed higher data density: 96.8% first-24h creatinine coverage and 89.2% 7-day AKI computability.
eICU showed lower density: 84.6% first-24h creatinine coverage and only 65.6% 7-day AKI computability.
Laboratory coverage gaps were significant; for example, eICU had much lower coverage for Magnesium (48.9% vs 93.8%), Phosphate (31.4% vs 90.7%), and Lactate (22.0% vs 53.7%) compared to MIMIC-IV.

2. Discrimination vs. Calibration:

Internal Performance: Both models achieved moderate discrimination (AUROC ~0.73).
External Discrimination: AUROC remained relatively stable during transport (MIMIC-IV $\to$ eICU: 0.703; eICU $\to$ MIMIC-IV: 0.722), suggesting rank ordering is preserved.
Calibration Shift: Significant directional shifts occurred:
- MIMIC-IV $\to$ eICU: The model overpredicted risk. Mean predicted risk (0.239) exceeded observed risk (0.185). The error was most severe in the highest-risk decile (+0.133).
- eICU $\to$ MIMIC-IV: The model underpredicted risk. Mean predicted risk (0.211) was lower than observed risk (0.269). The error was most prominent in mid-to-high risk strata (Decile 9 error: -0.180).

3. Failure-Mode Analysis:

Measurement Density: The MIMIC-IV model overpredicted risk most strongly in eICU stays with high measurement density ( $\ge$ 5 creatinine tests in 24h), suggesting the model conflated frequent testing (a proxy for clinical concern) with actual AKI risk.
Missingness: Strata with missing primary labs showed distinct error patterns, reinforcing that missingness profiles are a critical component of transportability risk.

Key Contributions

The EHR Data Passport: A practical framework that shifts the focus from "model performance" to "data readiness." It mandates checking endpoint computability, laboratory coverage, and measurement density before claiming a model is transportable.
Bidirectional Stress-Testing: By validating in both directions, the study demonstrates that transportability failure is not a one-way issue but depends on the specific data-generation processes of both source and target systems.
Calibration over Discrimination: The paper provides empirical evidence that high AUROC does not guarantee clinical utility in a new setting. It highlights that absolute risk calibration is the primary failure mode in cross-database AKI prediction.
Algorithmovigilance: The study operationalizes the concept of "algorithmovigilance," arguing that systematic monitoring of where and why a model fails (e.g., due to measurement intensity bias) is essential before deployment.

Significance and Claims

The paper does not claim to have built the most accurate AKI predictor. Instead, its significance lies in reframing renal-risk AI as a clinical systems problem.

Modest Claim: The authors assert that creatinine-based AKI prediction is feasible across these databases, but transportability is conditional. It depends on the alignment of endpoint computability, laboratory coverage, and measurement density.
Core Argument: Before a hospital adopts an AKI model, they must ask if their EHR can reproduce the label, predictor window, and missingness structure of the development set. If the "data passport" reveals significant gaps (e.g., lower creatinine frequency or different missingness patterns), the model is not ready for deployment, regardless of its AUROC.
Implication: The paper advocates for a "calibration-focused algorithmovigilance" approach. It suggests that external validation should not stop at discrimination metrics but must include calibration deciles and failure-mode analyses to ensure that risk estimates are clinically actionable and not merely rank-ordered.

In summary, the paper concludes that model validity is inextricably linked to data validity. A renal-risk AI system requires an accompanying EHR data passport and rigorous calibration checks to ensure it does not amplify alarm burdens or miss high-risk patients when moved across different healthcare environments.