📊 statistics

Compatibility of Missing Data Handling Methods across the Stages of Producing Clinical Prediction Models

This paper demonstrates that to ensure unbiased predictive performance and compatibility across the clinical prediction model lifecycle, researchers must align their missing data handling strategies during development and validation with the specific constraints of model deployment, such as using multiple imputation when missing data is excluded or maintaining consistent imputation methods when missing data is permitted.

Original authors: Antonia Tsvetanova, Matthew Sperrin, David A. Jenkins, Niels Peek, Iain Buchan, Stephanie Hyland, Marcus Taylor, Angela Wood, Richard D. Riley, Glen P. Martin

Published 2026-02-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Antonia Tsvetanova, Matthew Sperrin, David A. Jenkins, Niels Peek, Iain Buchan, Stephanie Hyland, Marcus Taylor, Angela Wood, Richard D. Riley, Glen P. Martin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to create the perfect recipe for a new dish (a Clinical Prediction Model). Your goal is to predict how a customer will react to the dish. To do this, you need specific ingredients (predictors like age, blood pressure, etc.). But sometimes, you don't have all the ingredients when you are cooking, or your customers might not have provided all the information about their dietary preferences.

This paper is about how to handle those "missing ingredients" at three different stages of your cooking process:

Development: Creating the recipe in your test kitchen.
Validation: Tasting the recipe with a new group of people to see if it works.
Deployment: Serving the dish to the real public.

The authors discovered that the way you handle missing ingredients in your test kitchen and your tasting session must match how you plan to serve the dish to the public. If you don't match them, your recipe will fail, or you will think it tastes great when it actually tastes terrible.

Here is the breakdown of their findings using simple analogies:

The Two Main Rules of the Game

The authors propose two golden rules for handling missing data:

The "No Degradation" Rule: When you develop your recipe, you must handle missing ingredients in a way that ensures the final dish tastes just as good as the "perfect" version (where you had all ingredients).
The "Truth-Telling" Rule: When you validate (test) your recipe, you must handle missing ingredients in a way that gives you an honest score. You don't want to trick yourself into thinking the recipe is better or worse than it really is.

Scenario A: The "Strict" Restaurant (No Missing Data Allowed)

Imagine a fancy restaurant where the chef refuses to cook a dish unless the customer provides every single ingredient on the menu. If a customer is missing an ingredient, the chef says, "Sorry, I can't cook for you."

The Paper's Advice: If your plan is to be this strict chef, you must develop and test your recipe using Multiple Imputation (MI).
- The Analogy: Think of Multiple Imputation as a "magic guesser" that fills in the missing ingredients based on what you do know, but it does this in a way that accounts for uncertainty. Crucially, this "magic guesser" needs to know the final outcome (did the customer like the dish?) to make good guesses during the development phase.
- The Trap: If you develop your recipe using "Mean Imputation" (just guessing the average ingredient for everyone) or "Complete Case Analysis" (throwing away any customer who missed an ingredient), your recipe will be weak. When you finally try to serve it to the strict public, it will taste bad (model degradation).

Scenario B: The "Flexible" Food Truck (Missing Data Allowed)

Now, imagine a food truck where the chef is okay with missing ingredients. If a customer forgets to say if they are allergic to nuts, the chef just uses a standard "default" nut level or a specific "guessing" method to fill it in so the cooking can continue.

The Paper's Advice: If your plan is to be this flexible food truck, you must use the exact same method to fill in the missing ingredients during development, testing, and serving.
- The Analogy: If you decide to use a "Regression Imputation" (a specific type of smart guess based on other ingredients) to fill in the blanks for your customers, you must use that same "smart guess" method when you are creating the recipe and when you are testing it.
- The Trap: If you create the recipe using "Mean Imputation" (averages) but then tell your customers, "Don't worry, we'll use a 'Smart Guess' method to fill in your missing info," the recipe will break. The math won't line up. The paper found that mixing methods (like developing with one method and validating with another) almost always leads to a biased score—you think your food truck is successful, but it's actually failing.

The "Pattern Sub-Model" (The Special Case)

The paper also looked at a method called Pattern Sub-Models.

The Analogy: Instead of one big recipe, you write four different mini-recipes:
1. One for customers who gave all info.
2. One for customers missing ingredient A.
3. One for customers missing ingredient B.
4. One for customers missing both.
The Paper's Advice: If you plan to use this "mini-recipe" approach at the food truck, you must develop and test using this exact same "mini-recipe" approach. You cannot develop with mini-recipes and then test by averaging everything together.

The Big Takeaway

The authors ran thousands of computer simulations and checked real-world surgery data to prove their point. They found that common practices are often wrong.

Common Mistake: Many researchers develop a model using one method (like Multiple Imputation) to get the best stats, but then deploy it using a simpler method (like Mean Imputation) because it's easier to code into an app.
The Result: This creates a "liar's paradox." The model looks great in the lab, but when it hits the real world, it performs poorly, or the performance scores are completely wrong.

Summary in One Sentence

To build a reliable medical prediction tool, you must decide first how you will handle missing data when the tool is actually used in the real world, and then you must use that exact same method to build and test the tool; mixing and matching methods leads to broken models and false confidence.

Technical Summary: Compatibility of Missing Data Handling Methods across the Stages of Producing Clinical Prediction Models

Problem Statement
Missing data in predictor variables is a pervasive challenge throughout the lifecycle of Clinical Prediction Models (CPMs), spanning development, validation, and deployment. While methodological research has extensively addressed missing data for causal inference (focusing on unbiased parameter estimates), the objectives of prediction modeling differ: the primary goal is accurate, well-calibrated predictions in new individuals. Current practice often lacks guidance on how missing data handling decisions in development and validation should align with the intended deployment strategy. Specifically, it is unclear whether the methods used to handle missingness during model building and testing are "compatible" with how the model will eventually be used in clinical practice. Inconsistencies—such as using Multiple Imputation (MI) during development but Mean Imputation during deployment—may lead to biased performance estimates or model degradation.

Methodology
The authors investigated this issue through two complementary approaches: a large-scale simulation study and an empirical analysis of real-world surgical data.

Simulation Study:
- Design: Following the ADEMP framework, the study simulated a target population of 200,000 individuals with two predictors ( $X_1, X_2$ ) and a binary outcome. $X_1$ was subject to missingness, while $X_2$ and the outcome were fully observed.
- Missingness Mechanisms: Six Directed Acyclic Graphs (DAGs) were used to induce Missing Completely at Random (MCAR), Missing at Random (MAR), and Missing Not at Random (MNAR) mechanisms. Missingness proportions ranged from 10% to 50%, and correlations between predictors were varied.
- Handling Methods: Five strategies were evaluated at each stage (development, validation, deployment):
  - Complete Case Analysis (CCA)
  - Mean/Mode Imputation
  - Single Deterministic Regression Imputation (RI)
  - Multiple Imputation (MI), tested both with the outcome included (MI-with-Y) and excluded (MI-no-Y).
  - Pattern Sub-Models (PSM).
- Compatibility Criteria: The study defined "compatibility" based on two principles:
  1. Minimizing Model Degradation: The difference between the deployed model's performance and the true model's performance under a specific deployment strategy.
  2. Unbiased Validation: The validation process must yield an unbiased estimate of the model's performance when deployed under that same strategy.
- Scenarios: The study tested all combinations of development, validation, and deployment strategies across 600 scenarios.
Empirical Analysis:
- Data: The authors utilized the North-West Clinical Outcomes Research Registry (NCORR) containing 6,600 patients undergoing lung resection surgery.
- Task: Predicting 90-day post-surgery mortality using a logistic regression model based on the RESECT-90 CPM.
- Procedure: Bootstrap sampling was used to create 100 development datasets. Models were developed using the various missing data handling methods and then validated against the original dataset, applying corresponding validation strategies to assess performance bias.

Key Results
The simulation and empirical results consistently demonstrated that compatibility is highly dependent on the intended deployment strategy:

Deployment Requiring Complete Data (E-all):
- To minimize degradation and ensure unbiased validation, models should be developed and validated using Multiple Imputation including the outcome (MI-with-Y) or fully observed data.
- Under complex missingness mechanisms (e.g., MNAR), even MI-with-Y showed some degradation, but it remained superior to other methods.
- Using CCA or Mean Imputation for development led to significant model degradation when deployed under an E-all strategy.
Deployment Allowing Missing Data:
- When a CPM is intended to be deployed with missing data allowed (using a specific imputation method like Mean Imputation, RI, MI-no-Y, or PSM), the same method must be used during both development and validation.
- Mean/Mode Imputation: Developing with MI and deploying with Mean Imputation resulted in biased performance estimates and calibration errors.
- Regression Imputation (RI) & PSM: These methods performed well only when the development, validation, and deployment strategies were identical.
- MI-no-Y: While theoretically appealing for deployment, using MI-no-Y during development led to biased estimates of the predictor-outcome association, causing degradation when deployed under an E-MI strategy.
Validation Bias:
- A critical finding was that validation bias was driven primarily by the method used to handle missing data during validation, rather than the method used during development.
- If a model is validated using a method different from the intended deployment strategy, the performance estimates will be biased, regardless of how the model was originally built.

Significance and Claims
The paper argues that the decision of how to handle missing data at deployment must fundamentally inform decisions made during development and validation. The authors claim that:

Current Practices are Flawed: Common combinations of methods found in literature (e.g., developing with MI and validating/deploying with Mean Imputation) frequently result in biased predictive performance estimates and sub-optimal models.
Alignment is Essential: To achieve compatibility, researchers must work backward from the deployment strategy.
- If the goal is a model for patients with complete data, use MI-with-Y (or complete data) throughout.
- If the goal is a model for patients with missing data, use the identical imputation method (e.g., RI or PSM) across development, validation, and deployment.
Practical Implications: The authors emphasize that while MI-with-Y is statistically robust for complete-data deployment, it is often impractical for deployment (requiring access to original data or complex computational resources). Therefore, if a model must handle missing data in practice, developers should prioritize deterministic methods like Regression Imputation or Pattern Sub-Models from the outset, rather than trying to adapt a model built for complete data.

The study concludes that without this alignment, the translation of CPMs into clinical practice risks being based on inaccurate performance estimates, potentially compromising patient care.