📊 statistics

Formalising Sample Size Calculations for the Development of Risk Prediction Models: The Importance of Accounting for Performance Variability

This paper proposes a new framework for sample size calculations in risk prediction model development that moves beyond targeting expected performance metrics to explicitly account for performance variability, thereby ensuring a high probability of achieving acceptable model calibration and robustness.

Original authors: Menelaos Pavlou, Rumana Z. Omar, Gareth Ambler

Published 2026-07-16

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Menelaos Pavlou, Rumana Z. Omar, Gareth Ambler

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to create the perfect recipe for a new dish. You have a list of ingredients (predictors) and you want to cook up a recipe (a model) that tastes exactly right for everyone who tries it. In the world of medicine, these "recipes" are called risk prediction models. They help doctors guess how likely a patient is to have a specific health event, like a heart attack or an infection, based on things like their age, weight, and medical history. But here is the tricky part: just because a recipe tastes amazing in your own kitchen doesn't mean it will taste the same in every other kitchen. If you only cook a few batches to test your recipe, you might get lucky and hit the perfect flavor by chance, or you might get a weird, salty batch just because you didn't mix enough ingredients. This is the problem of "variability." Scientists need to know exactly how many batches (or patients) they need to test to be sure their recipe is reliable, not just a lucky fluke.

For a long time, the standard way to figure out how many patients are needed was to aim for an "average" perfect result. It was like saying, "If we cook 1,000 batches, the average taste should be delicious." But the authors of this paper, Menelaos Pavlou, Rumana Z. Omar, and Gareth Ambler, realized that aiming for the average isn't enough. Sometimes, even if the average is great, you might end up with a batch that is completely inedible just because of bad luck with the ingredients. They argue that we need to change the goal: instead of just hoping the average is good, we should aim to make sure that most of the batches we cook are actually good. They call this the "Probability of Acceptable Performance." It's like saying, "We need to cook enough batches so that there is an 80% chance that any single batch we pull out of the oven will taste great," rather than just hoping the average of all batches is great.

The paper suggests that the old way of calculating sample sizes often leads to models that are unstable, especially when the model has only a few ingredients (predictors). If you try to bake a cake with just two ingredients, you need a much larger kitchen (more data) to be sure you aren't just getting lucky with the mixing. The authors used computer simulations to show that when you ignore this "variability," you might end up with a model that looks good on paper but fails in the real world. They propose a new method that accounts for this risk, often requiring larger sample sizes to ensure the model is robust. They also looked at a technique called "shrinkage," which is like gently adjusting your recipe to be a bit more conservative, and found that while it helps, it doesn't completely fix the problem if you don't have enough data to begin with.

The Recipe for a Reliable Prediction

Think of building a medical prediction model like training a robot to recognize cats in photos. You show the robot thousands of pictures, and it learns the patterns. If you only show it ten pictures, it might memorize those specific ten cats perfectly but fail completely when it sees a new cat. This is called "overfitting," and it's the enemy of good prediction. To avoid this, scientists need to know exactly how many pictures (patients) to show the robot.

The paper starts by explaining that the current rulebook for deciding how many patients are needed focuses on the expected value. Imagine you are rolling a die. The "expected value" is the average you get if you roll it a million times (which is 3.5). The old method says, "Let's roll the die enough times so that the average is 3.5." But the authors point out a flaw: if you only roll the die ten times, you might get a 6 every single time, or a 1 every single time. The average might be 3.5 in the long run, but your specific ten rolls could be a disaster. In medical terms, this means a model might look perfect on average across many studies, but the specific model built for your hospital might be wildly inaccurate.

The authors introduce a new concept called PrAP (Probability of Acceptable Performance). Instead of just caring about the average, PrAP asks: "What are the odds that the model we build right now will be good enough?" They define a "good enough" range, like a calibration slope between 0.85 and 1.15 (where 1 is perfect). The goal is to pick a sample size large enough so that there is, say, an 80% chance the model falls into that "good" zone.

The Simulation Kitchen

To test their idea, the authors didn't just guess; they ran thousands of computer simulations. They created fake patient data with different numbers of "ingredients" (predictors), ranging from 4 to 28. They compared two ways of deciding how many patients to use:

The Standard Way: Calculate the size needed so the average performance is 0.9 (close to perfect).
The New Way: Calculate the size needed so there is an 80% chance the performance is between 0.85 and 1.15.

The results were eye-opening. When the models had a small number of predictors (like 4 or 5), the Standard Way suggested a relatively small sample size. However, when they actually ran the simulations, they found that the models built with this small sample were all over the place. Sometimes they were great, but often they were terrible. The probability of getting a "good" model was only about 54%—basically a coin flip!

In contrast, the New Way demanded a much larger sample size. For a model with only 4 predictors, the New Way required 2.7 times more patients than the Standard Way. But the payoff was huge: with this larger sample, the probability of getting a good model jumped to the target 80%. The authors found that for models with many predictors (like 18 or more), the two methods suggested similar sample sizes, but for the smaller, simpler models, the difference was massive.

The "Shrinkage" Shortcut?

The paper also investigated a popular trick called shrinkage. Imagine you baked a cake and it was a little too sweet. Shrinkage is like taking a tiny bit of the sugar out to make it safer. In statistics, it involves slightly reducing the strength of the model's predictions to prevent overconfidence.

The authors tested if using shrinkage could let researchers get away with smaller sample sizes. They found that shrinkage did help improve the average performance, making the models closer to perfect. However, it also introduced a new kind of wobble. Because you have to estimate how much to shrink, that estimation adds its own uncertainty. The simulations showed that while shrinkage improved the odds of a good model, it didn't fix the problem entirely if the sample size was too small. In fact, for very small models (fewer than 6 predictors), the extra uncertainty from shrinkage meant the probability of a good model didn't improve much. The authors suggest that shrinkage is a helpful tool, but it shouldn't be used as a magic wand to skip collecting enough data.

A Real-World Test: Heart Valve Surgery

To prove their theory wasn't just a computer fantasy, the authors applied their methods to a real dataset of 16,679 patients who had heart valve surgery. They wanted to build a model to predict the risk of dying in the hospital.

Using the Standard method, they calculated they needed 2,250 patients. Using their new PrAP method, they calculated they needed 2,740 patients. They then ran a simulation where they repeatedly sampled these groups from the real data to see what happened.

The results confirmed their suspicions. The group with 2,250 patients (Standard) had a median performance that was good, but the models varied wildly. Some were great, some were poor. The group with 2,740 patients (New) was much more consistent. Almost all the models built with this larger group fell into the "acceptable" range. Furthermore, when they added shrinkage to the mix, the larger group (2,740) saw a clear improvement in reliability, while the smaller group didn't see much benefit.

The Takeaway

The main message of this paper is a warning against being too comfortable with "average" results. In the high-stakes world of medical prediction, an average success rate isn't enough if individual models can fail spectacularly. The authors suggest that to build truly reliable risk models, especially those with fewer ingredients, researchers need to be willing to collect more data. They propose a new way of calculating sample sizes that prioritizes the probability of success over the average success.

While this might mean gathering more patients than previously thought necessary, the trade-off is a model that is stable and trustworthy. As the authors note, if you want to be 80% sure your model works, you can't just aim for the average; you have to aim for the safety margin. Their work provides the mathematical tools to do exactly that, ensuring that when a doctor relies on a prediction model, it's not just a lucky guess, but a well-tested recipe.

Technical Summary: Formalising Sample Size Calculations for the Development of Risk Prediction Models

1. Problem Statement

Current methodologies for calculating sample sizes in the development of risk prediction models primarily focus on ensuring that the expected value of a performance measure (e.g., calibration slope, C-statistic) meets a prespecified target. For instance, standard approaches often select a sample size such that the expected calibration slope (CS) equals 0.9. However, these methods neglect sampling variability. Consequently, even when adhering to recommended sample sizes, the actual performance of a model developed on a specific dataset can vary substantially. When variability is high, the probability of obtaining a model with performance close to the target may be unacceptably low, leading to model instability and unreliable individual predicted probabilities. This issue is particularly pronounced when the number of predictors is small relative to the sample size required for average performance targets.

2. Methodology

The authors propose a framework that explicitly incorporates both the expected value and the variability of performance measures into sample size calculations. The methodology is grounded in the following components:

Data-Generating Mechanism (DGM): The framework assumes a joint distribution $P(Y, \mathbf{X}; \boldsymbol{\phi})$ for the outcome $Y$ and predictors $\mathbf{X}$ . While the primary focus is on binary outcomes modeled via logistic regression with Maximum Likelihood Estimation (MLE), the principles are presented generally.
Probability of Acceptable Performance (PrAP): A new metric is introduced to quantify the likelihood that a model's performance falls within a prespecified acceptable range (e.g., $0.85 \leq \text{CS} \leq 1.15$ ). Sample sizes are then determined to ensure this probability exceeds a target threshold (e.g., 80%).
Bias-Correction for Analytical Formulas: The paper addresses a known limitation in existing analytical formulas (e.g., Riley et al., 2019) which systematically underestimate sample sizes for models with high predictive strength (high C-statistic). The authors propose a heuristic adjustment where the anticipated C-statistic ( $C$ ) is replaced by a conservative adjusted value ( $C_{adj}$ ). This adjustment accounts for the discrepancy between logistic regression coefficients and linear discriminant analysis (LDA) coefficients under high discrimination, effectively inflating the required sample size to meet calibration targets.
Analytical Approximation for PrAP: To avoid the computational intensity of full simulation, the authors derive an approximate analytical formula for the variance of the calibration slope ( $\text{Var}(\hat{s}_n)$ ). This formula, combined with the expected calibration slope, allows for the calculation of PrAP and the subsequent determination of the required sample size via numerical optimization.
Evaluation of Shrinkage: The study investigates the impact of post-estimation linear shrinkage (and penalized methods like Ridge regression) on PrAP. Simulations and a real-data case study assess whether shrinkage can mitigate the need for larger sample sizes or improve the probability of acceptable performance.

3. Key Contributions

Formalisation of PrAP: The paper formally defines the "Probability of Acceptable Performance" as a primary target for sample size calculations, shifting the focus from average performance to the stability of performance across development datasets.
Demonstration of Variability Impact: Through simulation studies, the authors demonstrate that standard calculations (targeting median CS = 0.9) result in unacceptably low PrAP for models with a small number of predictors. For example, with $p=4$ predictors, the probability of achieving a CS between 0.85 and 1.15 was only 54% under standard calculations, whereas the new approach (targeting PrAP = 0.8) required significantly larger sample sizes to ensure stability.
Heuristic Adjustment for High Discrimination: The paper provides a practical correction ( $C_{adj}$ ) for existing analytical formulas, improving their accuracy for models with high C-statistics ( $C \geq 0.8$ ).
Analytical Approximation: An efficient analytical method is proposed to approximate the sample size required for a target PrAP, offering a computationally lighter alternative to simulation-based optimization while maintaining reasonable accuracy.
Role of Shrinkage: The study clarifies that while shrinkage methods (e.g., linear shrinkage factor, Ridge) generally improve the median calibration slope, they introduce additional variability due to the estimation of the shrinkage factor. Consequently, shrinkage does not universally allow for reduced sample sizes; in small datasets, the increased variability can attenuate the benefits of shrinkage regarding PrAP.

4. Results

Simulation Findings:
- Standard sample size calculations (targeting median CS = 0.9) yield large variability in performance for small models (few predictors), leading to low PrAP.
- The proposed "New" calculation (targeting PrAP = 0.8) results in substantially larger sample sizes for small models (e.g., 2.7 times larger for $p=4$ ) but converges with standard calculations for larger models ( $p \geq 18$ ).
- The proposed bias-correction ( $C_{adj}$ ) significantly improves the accuracy of analytical formulas for high C-statistics, bringing the median calibration slope closer to the target of 0.9.
- The analytical approximation for PrAP closely matches simulation-based results, except at very high discrimination levels ( $C > 0.85$ ).
- Shrinkage improves the probability of acceptable calibration for models with a moderate-to-high number of predictors ( $p > 6$ ) but offers limited benefit for very small models where the uncertainty in the shrinkage factor dominates.
Case Study (Heart Valve Surgery):
- Applied to a dataset of 16,679 patients, the new calculation increased the required sample size from 2,250 (standard) to 2,740 (new) to achieve PrAP = 0.8.
- Resampling confirmed that the new calculation achieved the target PrAP, whereas the standard calculation did not.
- Shrinkage methods (Linear Shrinkage Factor, Modified Ridge) improved calibration performance and PrAP compared to MLE alone, consistent with simulation findings.

5. Significance and Claims

The paper argues that accounting for performance variability is essential for robust model development. The authors claim that relying solely on expected performance targets can lead to the development of unstable models, particularly in scenarios with few predictors. By introducing the PrAP framework, the paper provides a mechanism to ensure that a model developed on a specific dataset is likely to perform acceptably upon validation.

The authors modestly claim that their proposed framework and tools (implemented in the R package samplesizedev) offer a more rigorous approach to sample size determination. They emphasize that while simulation-based methods are computationally intensive, the proposed analytical approximations provide a viable and efficient alternative. Furthermore, they caution against using shrinkage as a primary strategy to reduce sample size requirements, suggesting instead that sample sizes should be calculated assuming MLE, with shrinkage applied during model fitting to improve performance, provided the number of predictors is not extremely small. The work underscores the necessity of moving beyond "average" performance metrics to ensure the reliability of risk prediction models in clinical practice.