Unmeasured but Not Unbiased: The Missingness Demographic Leakage Audit (MDLA) for Calibration-Aware Fairness Evaluation in Critical Care Mortality Prediction

This paper introduces the Missingness Demographic Leakage Audit (MDLA), a reproducible framework that reveals how patterns of missing clinical data in critical care mortality models can act as subtle, unmeasured demographic proxies, necessitating the integration of missingness-aware auditing and calibration-aware evaluation into clinical AI validation pipelines.

The Gist

Imagine you are trying to predict who might get sick in a hospital's intensive care unit (ICU) using a computer program. You feed the program data like heart rate, blood pressure, and lab results. Usually, when researchers check if this program is "fair," they look at the numbers it does see. They ask: "Does the program make the same mistakes for Black patients as it does for White patients?"

But this paper points out a huge blind spot. It asks a different question: "What does the program learn from the numbers that are missing?"

Here is the story of the paper, broken down into simple concepts and analogies.

1. The "Silent Clue" (The Problem)

Imagine you are trying to guess someone's background just by looking at their grocery list.

• The Obvious Way: You look at what they bought (e.g., "They bought kale, so they might be health-conscious").
• The Hidden Way: You look at what they didn't buy. Maybe they never bought a specific type of expensive meat because their local store doesn't stock it, or because of how much money they have.

In the ICU, doctors order tests (like blood gases) for patients. Sometimes, a test is missing.

• Standard View: "Oh, the test is missing. Let's just guess the value or ignore it."
• This Paper's View: "Wait! The fact that the test is missing might actually be a secret clue about the patient's race or insurance status."

The authors found that in their data, certain tests were missing much more often for Black patients than for White patients. It wasn't random; it was a pattern. The computer program, if it's smart enough, can accidentally learn to use these "missing" patterns as a shortcut to guess a patient's race, even if you never told it the patient's race.

2. The Detective Tool: MDLA

To catch this "silent clue," the authors built a new tool called MDLA (Missingness Demographic Leakage Audit). Think of this as a metal detector for hidden bias.

Instead of just checking the final answer the computer gives, MDLA checks the "footprints" left behind by missing data.

• Step 1: They created a list of "Missing Flags" (like a checklist where a checkmark means "This test was skipped").
• Step 2: They asked a simple computer model: "Can you guess a patient's race just by looking at this checklist of missing tests?"
• The Result: Yes! The model could guess the race better than flipping a coin. This proved that the absence of data carries demographic information.

3. The "Aha!" Moment: The Computer is Using the Clue

The most important part of the paper is what happens when they let the main prediction model see these "Missing Flags."

• The Experiment: They trained a model to predict death risk. First, they gave it only the real numbers (heart rate, etc.). Then, they gave it the real numbers plus the "Missing Flags."
• The Surprise: When the model was allowed to see the "Missing Flags," the gap in performance between different racial groups got worse.
• The Analogy: Imagine a student taking a test. If they are allowed to peek at a cheat sheet that says "If the teacher didn't ask Question 5, the student is likely from Group A," the student might start guessing based on that instead of the actual math. The paper found that the computer was doing exactly this: it was using the "missing test" patterns as a shortcut, which made the predictions less fair for certain groups.

4. Fixing the "Broken Thermometer" (Calibration)

The paper also looked at how "confident" the computer was in its answers.

• The Problem: Sometimes the computer says, "There is a 20% chance of death," but for Black patients, the actual death rate might be 30%. The computer is "miscalibrated" for that group. It's like a thermometer that always reads 5 degrees too low for one specific room.
• The Solution: The authors tried different ways to "recalibrate" the computer. They found that a simple fix called Global Platt Scaling worked best.
• The Result: This simple fix made the computer's confidence much more accurate (reducing errors by 94%) without making the overall predictions worse. It's like adjusting the thermometer so it reads the right temperature for everyone, without needing to build a whole new thermometer.

5. The Big Takeaway

The paper concludes with a clear message for anyone building or using these hospital AI tools:

"Missing data is not just a mistake; it's a message."

If you ignore the fact that certain tests are missing more often for certain groups, your AI might be secretly using those gaps to make unfair decisions. Before you let an AI help make life-or-death decisions in a hospital, you need to run a "Missingness Audit" (like the MDLA tool) to make sure the computer isn't relying on these hidden, unfair shortcuts.

In short: The paper didn't just find a bug; it found a whole new way bugs can hide (in the empty spaces of the data) and gave doctors a new checklist to find them before they cause harm.

Technical Summary

1. Problem Statement

Clinical prediction models trained on Electronic Health Records (EHR) are routinely audited for fairness based on observed feature values. However, a critical blind spot exists: standard audits ignore the patterns of missing data. In critical care, the absence of measurements (e.g., laboratory values, vital signs) is often not random (Missing Not At Random - MNAR). Instead, missingness patterns can reflect structural inequities, differences in ordering practices, and resource availability tied to race, insurance status, and socioeconomic position.

The authors hypothesize that these "absence patterns" function as latent demographic proxies. If machine learning models inadvertently learn from these missingness indicators, they may encode demographic bias without explicitly using demographic variables, a pathway invisible to standard fairness audits.

2. Methodology

The study introduces the Missingness Demographic Leakage Audit (MDLA), a reproducible four-step informatics framework, and applies it to ICU mortality prediction.

• Datasets:
  • Development: MIMIC-IV v2.2 (n=50,827 ICU stays, single-center, Boston).
  • External Validation: eICU-CRD v2.0 (n=137,773 ICU stays, multi-center, 208 US hospitals).
  • Total cohort: ~188,600 patient-stays.
• Feature Engineering:
  • 43 clinical features (20 vital signs, 21 labs, 2 composites).
  • 44 Binary Missingness Indicators: Created for every feature (including excluded ones like troponin) where miss_X = 1 if the value was absent.
• MDLA Framework (4 Steps):
  1. Indicator Construction: Creating binary flags for missingness prior to imputation.
  2. Demographic Predictability Testing: Training a multinomial logistic regression solely on the 44 missingness indicators to predict race/ethnicity and insurance type.
  3. Feature-Level Association: Chi-square tests (Bonferroni-corrected) to link specific missingness indicators to demographic groups, quantified by Cramér's V.
  4. Model Reliance Ablation: Retraining XGBoost models with and without the missingness indicators to measure the change in racial AUROC disparity.
• Fairness & Calibration Audit:
  • Evaluated five criteria: Equalized Odds, Equal Opportunity, Demographic Parity, Predictive Parity, and Calibration Fairness (Expected Calibration Error - ECE).
  • Tested six post-hoc recalibration strategies (Global Platt, Group-Conditional Platt, Isotonic, Fairness-aware Temperature Scaling) on a Fairness-Utility Pareto Frontier.
• Standards: Followed TRIPOD+AI guidelines; all preprocessing parameters were derived from the development set and applied without modification to external validation.

3. Key Contributions

1. MDLA Framework: A structured protocol to detect "missingness-encoded" demographic signals, proving that absence patterns carry statistically significant demographic information.
2. Ablation Evidence: Demonstrated that adding missingness indicators to models increases racial performance disparity without improving global accuracy, confirming models rely on these subtle bias pathways.
3. Calibration-Centric Solution: Identified that Global Platt Scaling is the optimal post-hoc strategy, achieving massive calibration improvements with zero loss in discrimination, and successfully transferring to external validation without retraining.
4. Empirical Validation: Validated findings across two distinct, institutionally diverse databases (single-center vs. multi-center) totaling nearly 190,000 patients.

4. Key Results

A. Missingness as a Demographic Proxy (MDLA Steps 1–3)

• Predictability: Missingness patterns alone predicted racial group membership with an AUROC of 0.543 (95% CI: 0.540–0.546) and insurance type with AUROC 0.534.
• Significant Features: 18 out of 43 features showed Bonferroni-significant associations with race.
• Specific Gaps: Arterial blood gas measurements (PaCO₂, PaO₂, pH) showed the largest racial missingness gaps (up to 14 percentage points; e.g., Black patients had 51.6% missingness vs. 47.9% for White patients).
• Effect Size: While statistically significant, effect sizes were small (all Cramér's V < 0.10), indicating a subtle but pervasive signal.

B. Model Reliance (MDLA Step 4)

• Ablation Impact: When XGBoost was trained with missingness indicators added to the feature set, the racial AUROC disparity increased by 10.7% (from 0.063 to 0.069).
• Global Performance: Global AUROC remained stable (0.910 vs. 0.912), proving the model was weighting demographic signals via missingness rather than improving clinical prediction.

C. Fairness & Calibration Performance

• Internal Performance: XGBoost achieved AUROC=0.910. However, it exhibited significant fairness gaps, with an Equalized Odds TPR gap of 0.214 between racial groups.
• External Validation: The model suffered a generalization gap (AUROC dropped to 0.799) and severe miscalibration (ECE=0.377).
• Impossibility Theorem: The study confirmed that differing baseline mortality rates across subgroups make it mathematically impossible to simultaneously satisfy Equalized Odds, Predictive Parity, and Calibration Fairness. The authors argue Calibration Fairness is the clinically critical criterion for risk prediction.

D. Recalibration Strategies

• Global Platt Scaling: Reduced overall calibration error (ECE) by 94% (from 0.124 to 0.007) internally and by 69% externally, with zero AUROC loss.
• Group-Conditional Failure: Group-specific calibrators (by race) performed worse than global calibrators due to overfitting in small minority subgroups (e.g., Hispanic n≈400, Asian n≈200).
• Temperature Scaling: Failed to improve calibration because the model's miscalibration was intercept-driven, whereas temperature scaling only adjusts logit spread.

5. Significance and Implications

• Redefining Bias Detection: The paper establishes that "missing data" is not merely a nuisance to be imputed but a potential upstream bias pathway. Standard fairness audits that only look at observed values miss this critical leakage.
• Clinical Deployment: For clinical AI, calibration (accurate absolute risk estimation) is more important than discrimination for decision-making. The study proves that simple, global recalibration (Platt scaling) can fix severe miscalibration without sacrificing fairness or performance, and it generalizes well to new hospitals.
• Policy & Governance: The authors recommend integrating Missingness-Aware Auditing and Calibration-Aware Evaluation as mandatory steps in clinical AI validation pipelines before deployment.
• Limitations: The study is retrospective; the missingness signal is subtle (requiring large sample sizes to detect); and external validation was limited by the lack of insurance data in the eICU database.

Conclusion: The study concludes that clinical measurement absence is a structured demographic signal. The MDLA framework provides a necessary tool to detect this, and Global Platt scaling offers a robust, deployable remedy to ensure models are calibrated fairly across diverse populations.