📊 statistics

Incorporating Missing Data Considerations into Sample Size Calculations for Developing Clinical Prediction Models

This study demonstrates that missing predictor data significantly increases the sample size requirements for developing stable and well-calibrated clinical prediction models, and proposes a practical adaptation of posterior-based sample size calculations to directly incorporate missing data assumptions and imputation strategies.

Original authors: Glen P. Martin, Sian Bladon, Rebecca Whittle, Molly Wells, Gary S. Collins, Richard D. Riley

Published 2026-05-11

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Glen P. Martin, Sian Bladon, Rebecca Whittle, Molly Wells, Gary S. Collins, Richard D. Riley

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Why We Need More Data When Things Go Missing

Imagine you are a chef trying to create a new, perfect soup recipe (a Clinical Prediction Model). To make sure the soup tastes right for everyone, you need to taste-test it on a large group of people.

Usually, statisticians have a "rule of thumb" for how many people you need to taste-test to get a reliable recipe. They assume that every single person you ask will have all the ingredients ready to go (e.g., they all have salt, pepper, and carrots).

The Problem: In the real world, people often forget to bring an ingredient. Maybe someone has no carrots, or their salt is missing. This is missing data.

This paper argues that if you expect people to be missing ingredients, your old "rule of thumb" for how many people to taste-test is wrong. It's too low. If you stick to the old number, your soup might taste great in your small test kitchen, but it will be a disaster when you try to serve it to the whole world.

The Core Discovery: Missing Ingredients Need a Bigger Pot

The researchers ran a massive simulation (a computer experiment) to see what happens when you try to cook with missing ingredients.

The "Perfect" Scenario: If everyone brings all their ingredients, the standard recipe for the number of testers works fine.
The "Missing" Scenario: When 20%, 40%, or even 60% of the testers are missing ingredients, the recipe starts to fail. The soup becomes "over-fitted."
- Analogy: Think of "over-fitting" like memorizing a specific test. If you only study the exact questions on one practice test, you might get 100% on that test. But if you walk into the real exam and the questions are slightly different (or some are missing), you fail. The model has learned the "quirks" of the small, messy dataset rather than the true pattern.

The Finding: To get the same reliable soup recipe when ingredients are missing, you often need twice as many testers as the standard rules suggest.

The Solution: A New "Simulation Kitchen"

The paper proposes a new way to figure out how many people you need. Instead of just doing a quick math calculation, they suggest running a simulation (a practice run) before you even start your real study.

Here is how their new method works, step-by-step:

Make a Fake World: First, you create a giant, perfect computer world of 500,000 people where you know exactly what the "true" soup recipe should be.
Break the World: Then, you intentionally "break" this world by making some people forget their ingredients (simulating missing data).
The Practice Run: You try to cook your soup using different strategies:
- Strategy A: Throw away anyone missing an ingredient (Complete Case Analysis).
- Strategy B: Guess the missing ingredient based on what they did bring (Imputation).
Check the Taste: You taste the soup. Does it match the "true" recipe?
Adjust the Pot Size: If the soup tastes bad, you increase the number of testers in your simulation and try again. You keep doing this until you find the exact number of testers needed to guarantee the soup will taste good, even with missing ingredients.

Key Concepts Explained Simply

Overfitting: Imagine a student who memorizes the answers to a specific practice quiz. They get an A. But when the real test comes, and one question is missing, they panic and fail. The model is "overfit" because it learned the noise (the missing data quirks) instead of the signal (the real pattern).
Calibration Slope: This is a score that tells you how "shrink-wrapped" your model is. A score of 1.0 is perfect. A score below 0.9 means your model is too confident and likely to be wrong. The paper found that with missing data, scores often dropped below 0.9 unless you doubled your sample size.
Imputation: This is the act of "guessing" the missing values. The paper tested different ways of guessing (like using a random forest or a simple average) and found that while guessing helps, it doesn't fix the problem entirely—you still need more data.
EVPI (Expected Value of Perfect Information): This is a fancy way of asking, "How much are we losing by not having perfect data?" It calculates the cost of uncertainty. The paper shows that missing data increases this "cost," and adding more people to your study lowers it.

What the Paper Actually Says (and Doesn't Say)

What they found: Missing data makes your model less stable and less accurate. To fix this, you need to increase your sample size, sometimes by a factor of 2.
What they recommend: Don't just use the old, simple math formulas. Use their new simulation method to plan your study. This involves making assumptions about how data goes missing (e.g., is it random, or does it happen because of other factors?) and testing those assumptions in a computer simulation.
What they didn't do: They did not test this on a specific real-world hospital right now. They used computer simulations and two hypothetical examples (one where they had old data to help them guess, and one where they had to guess blindly).
The "Rule of Thumb" Alternative: If you can't do the complex simulation, they suggest a rough fix: take your standard sample size number and divide it by (1 - percentage of missing data). For example, if you expect 50% missing data, you need double the sample size. However, they admit this is just a rough guess and the simulation is better.

The Bottom Line

If you are planning a study to build a medical prediction model, and you know that some data will be missing (which is almost always the case), do not trust the standard sample size calculators. They will give you a number that is too small.

Instead, use this new "simulation kitchen" approach. It allows you to test different scenarios and find the real number of people you need to recruit to ensure your model is stable, accurate, and ready for the real world.

Technical Summary: Incorporating Missing Data Considerations into Sample Size Calculations for Developing Clinical Prediction Models

Problem Statement
Clinical prediction models (CPMs) require sufficiently large datasets to minimize overfitting and ensure robust predictive performance, including accurate calibration and discrimination. Existing sample size calculation methods for CPM development (e.g., Riley et al., 2019; 2020) implicitly assume that all predictor data are completely observed for every participant. However, missing predictor values are ubiquitous in health data. If missing data are not accounted for during the design phase, the application of standard closed-form sample size calculations may result in an insufficient effective sample size. This discrepancy increases the risk of overfitting, model degradation (miscalibration, reduced discrimination), and loss of clinical utility. Currently, there is no established framework for integrating missing data assumptions and handling strategies directly into sample size calculations for CPM development.

Methodology
The authors propose an adaptation of the general simulation-based sample size framework originally developed by Riley et al. (2019), which utilizes anticipated posterior (sampling) distributions. The proposed extension explicitly incorporates missing data mechanisms and handling strategies into the data-generating process.

The adapted procedure involves the following steps:

Specification of Assumptions: Researchers must define the missingness mechanism (Missing Completely at Random [MCAR], Missing at Random [MAR], or Missing Not at Random [MNAR]), the expected patterns of missingness (which variables are missing jointly), and the anticipated proportion of missing data. These assumptions apply to both the development dataset and the target population.
Inducing Missingness: In the simulation, missing data are induced into the development dataset (and the target population if "pragmatic performance" is targeted) using fully conditional specification (e.g., via the ampute function in the mice R package).
Handling Missing Data: The planned imputation strategy (e.g., multiple imputation, random forest imputation, complete case analysis) is applied to the simulated development data. The CPM is then developed using the imputed data.
Evaluation: The developed model is evaluated on a large target population (either fully observed or imputed, depending on the deployment scenario). Performance metrics, including calibration slope, discrimination, and Expected Value of Perfect Information (EVPI), are calculated.
Iteration: The process is repeated across multiple iterations (e.g., 100) to generate posterior distributions of performance metrics. The development sample size ( $N_{dev}$ ) is increased iteratively until the desired performance targets (e.g., assurance probability of calibration slope between 0.9 and 1.1) are met.

To validate this approach, the authors conducted a simulation study following the ADEMP framework. They generated 240 scenarios varying predictor correlations, outcome prevalence, missingness mechanisms (MAR/MNAR), missingness proportions (10%–60%), and sample size multipliers ( $\delta$ ) relative to the closed-form minimum. They compared various imputation methods and modeling approaches (unpenalized, AIC-selected, and LASSO-penalized logistic regression).

Key Results

Underestimation by Closed-Form Methods: When development datasets contained missing values, the standard closed-form sample size calculations (assuming complete data) resulted in models with reduced predictive performance. Specifically, the median calibration slope frequently fell below the targeted value of 0.9, and assurance probabilities (the likelihood of achieving a slope between 0.9 and 1.1) dropped significantly (e.g., below 0.4 with 40–60% missingness).
Sample Size Inflation: To achieve performance comparable to models developed on fully observed data, the required sample size often needed to be inflated. In scenarios with high missingness (e.g., 60%), sample sizes up to twice the closed-form minimum were necessary to achieve stable calibration slopes and adequate assurance probabilities.
Impact of Imputation Methods: While different imputation methods (e.g., multiple imputation, random forest) showed varying degrees of performance degradation, none fully mitigated the risk of overfitting at the standard minimum sample size. Complete case analysis was particularly prone to overfitting.
Role of Penalization: LASSO penalization improved the average calibration slope across missing data scenarios, often keeping it above 0.9 even at the minimum sample size. However, it did not fully resolve the instability (low assurance probability) introduced by missing data, reinforcing the need for sample size adjustments.
EVPI Quantification: The simulation framework allowed for the calculation of the Expected Value of Perfect Information (EVPI), quantifying the loss in clinical utility due to finite sample sizes and missingness. Increasing the sample size reduced the gap in EVPI between models developed on imputed data versus fully observed data.

Practical Application
The authors illustrated the framework through two examples:

With Individual Participant Data (IPD): Using the MIMIC-III database to inform joint distributions and missingness patterns. The study demonstrated that at the closed-form minimum sample size ( $N=2836$ ), assurance probabilities were low (<45%). Increasing the sample size by 50% or 100% significantly improved calibration slopes and assurance probabilities.
Without IPD: A scenario where assumptions were derived from literature and expert knowledge rather than existing data. Similar trends were observed, confirming that the framework is applicable even when prior data is limited, provided assumptions are clearly articulated.

Significance and Claims
The paper claims to be the first investigation into how missing data assumptions and handling strategies should be incorporated into sample size calculations for CPM development. The authors argue that missing predictor data fundamentally increases the minimum sample size required to develop stable, well-calibrated, and clinically useful models.

They assert that their adapted simulation-based framework provides a "principled way" to determine minimum sample sizes that account for missingness directly, rather than relying on post-hoc adjustments or ignoring the issue. While they acknowledge that specifying missingness mechanisms can be challenging, they emphasize that all sample size calculations rely on assumptions; the key is to make these explicit.

The authors modestly suggest that in the absence of sufficient information to run the full simulation, a pragmatic alternative is to inflate closed-form sample size calculations by a factor of $1/(1 - \pi_{miss})$ , though they maintain that the simulation-based approach is preferred. They conclude that explicitly accounting for missing data during the study design phase is essential to avoid underpowered studies and ensure the reliability of clinical prediction models.