Bayesian sample size calculations for external validation studies of risk prediction models

1. Problem Statement

Current methods for calculating sample sizes in external validation studies of risk prediction models (specifically for binary outcomes) rely on frequentist approaches (e.g., the multi-criteria framework by Riley et al.). These methods have two primary limitations:

1. Fixed Assumptions: They require investigators to specify fixed, point-estimate values for model performance metrics (e.g., c-statistic, calibration slope) in the target population. This ignores the inherent uncertainty regarding true model performance derived from previous studies with finite samples.
2. Inappropriate Metrics for Clinical Utility: For Net Benefit (NB), a decision-theoretic measure of clinical utility, conventional precision-based inference (e.g., Confidence Interval widths) is often irrelevant. The goal of validation is often to determine the optimal clinical strategy, not just to estimate a parameter with a specific precision.

Consequently, existing methods may yield sample sizes that are either insufficient to guarantee desired precision in a specific dataset or fail to address the actual clinical decision-making value of the study.

2. Methodology

The authors propose a general Bayesian framework for sample size determination that treats model performance metrics as random variables rather than fixed constants.

A. Core Framework

• Prior Specification ($P(\theta)$): Instead of fixed values, the method requires specifying a prior distribution for the parameters $\theta$ governing the joint distribution of predicted risks ($\pi$) and outcomes ($Y$). The authors demonstrate that $\theta$ can be defined by outcome prevalence ($\phi$), the c-statistic ($c$), and the calibration function ($h$).
• Identifiability: Under mild regularity conditions (monotonicity of the calibration function and quantile identifiability of risk distributions), the joint distribution $P(\pi, Y)$ is fully identified by $\{\phi, c, h\}$.
• Pre-posterior Analysis: The framework utilizes Monte Carlo sampling to generate the "pre-posterior" distribution. This involves:
  1. Drawing $\theta$ from the prior $P(\theta)$.
  2. Simulating a validation dataset $D_N$ of size $N$ conditional on $\theta$.
  3. Calculating performance metrics (e.g., CI widths, NB) from $D_N$.
  4. Repeating this process to build the distribution of anticipated outcomes for a given $N$.

B. Sample Size Rules

The framework introduces three distinct classes of sample size rules:

1. Expected Precision (ECIW): Targets the expected value of the Confidence Interval (CI) width across the distribution of future datasets.
   • Rule: Find minimum $N$ such that $E[CI Width] \leq \tau$.
2. Assurance-Based Rules (QCIW): Targets the probability (assurance) that the CI width will meet a threshold. This accounts for the risk that a specific dataset yields poor precision even if the expected precision is good.
   • Rule: Find minimum $N$ such that $P(CI Width \leq \tau) \geq q$ (e.g., 90% assurance).
3. Decision-Theoretic Rules (for Net Benefit):
   • Optimality Assurance: The probability that the strategy identified as best in the sample is actually the best strategy in the population.
   • Value of Information (VoI):
     • EVSI (Expected Value of Sample Information): The expected gain in Net Benefit from conducting the study compared to current knowledge.
     • rEVSI (Relative EVSI): The ratio of EVSI to EVPI (Expected Value of Perfect Information), providing a unit-less metric of information gain.
     • ENBS (Expected Net Benefit of Sampling): Maximizes $EVSI \times \text{Population Size} - \text{Sampling Cost}$.

C. Implementation

The authors developed the bayespmtools R package. It uses:

• Monte Carlo Sampling: To simulate pre-posterior distributions.
• Stochastic Root-Finding: Algorithms (Robbins-Monro for CI widths; Stochastic-Simultaneous Optimistic Optimization for VoI) to find the minimum $N$ satisfying the rules.
• Two-step Approximation: An alternative method for CI width targets that avoids simulating individual-level data by applying Standard Error (SE) equations twice (once for parameter uncertainty, once for sampling variability).

3. Key Contributions

1. Uncertainty Quantification: Shifts sample size calculation from assuming fixed "truths" to explicitly modeling uncertainty in model performance metrics based on prior evidence (e.g., meta-analyses).
2. Assurance Probabilities: Introduces rules that guarantee a high probability of meeting precision targets, addressing the risk of "unlucky" datasets failing to meet criteria.
3. VoI Integration: Proposes the first sample size rules for Net Benefit based on Optimality Assurance and Value of Information, moving beyond arbitrary CI widths for clinical utility metrics.
4. Hybrid Approach: Demonstrates a flexible workflow where sample sizes can be determined by one criterion (e.g., calibration slope precision) and evaluated against others (e.g., VoI or Optimality Assurance) to optimize study design.
5. Software Tool: Provides an open-source R package (bayespmtools) to implement these complex stochastic calculations.

4. Case Study Results

The authors applied the framework to validate the ISARIC 4C model for predicting deterioration in hospitalized COVID-19 patients in the London region.

• Inputs: Meta-analytic data from 8 UK regions was used to construct predictive distributions for prevalence, c-statistic, mean calibration, and calibration slope.
• Comparison:
  • Conventional (Riley): Required N = 1,056 (driven by calibration slope precision).
  • Bayesian (Expected CI Width): Required N = 1,064.
  • Bayesian (90% Assurance): Required N = 1,181 (driven by calibration slope assurance).
• VoI Analysis:
  • The EVSI curve showed diminishing returns. Increasing the sample size from 522 (driven by c-statistic/NB assurance) to 1,181 (driven by calibration slope assurance) resulted in only an 8.8% gain in Net Benefit.
  • Calibration Error Plots: Visual inspection of calibration curves at $N=522$ showed that 95% credible intervals for calibration errors remained acceptable (<0.1) for most risk strata.
• Final Recommendation: By relaxing the strict calibration slope precision requirement in favor of VoI and clinical utility, the recommended sample size was reduced to N = 522, significantly lowering resource requirements without compromising clinical decision-making.

5. Significance and Implications

• Efficiency: The Bayesian approach can identify scenarios where conventional methods overestimate sample sizes, particularly when strict precision on statistical metrics (like calibration slope) yields minimal clinical value (VoI).
• Risk Management: Assurance-based rules allow investigators to explicitly manage the risk of obtaining ambiguous results, which is crucial for public funding and clinical adoption.
• Decision-Theoretic Alignment: By focusing on VoI and Optimality Assurance, the method aligns sample size design with the ultimate goal of validation: making better clinical decisions, rather than just achieving statistical precision.
• Cultural Shift: The paper advocates for a shift in the prediction modeling community toward explicitly characterizing and communicating uncertainty in model performance, moving away from deterministic "final model" reporting.
• Future Directions: The framework is extendable to survival outcomes, individual-level prediction uncertainty, and model updating scenarios, though it currently requires numerical integration and stochastic optimization which can be computationally intensive.

In summary, this paper provides a rigorous, flexible, and clinically relevant alternative to frequentist sample size calculations, enabling researchers to design validation studies that are robust to uncertainty and optimized for clinical utility.