📊 statistics

Adaptive Gaussian Process Search for Simulation-Based Sample Size Estimation in Clinical Prediction Models: Validation of the pmsims R Package

This paper introduces and validates the `pmsims` R package, which utilizes an adaptive Gaussian process search engine to provide a flexible and computationally efficient framework for simulation-based sample size estimation in clinical prediction models, outperforming existing analytical and non-adaptive methods across diverse outcome types.

Original authors: Oyebayo Ridwan Olaniran, Diana Shamsutdinova, Sarah Markham, Felix Zimmer, Daniel Stahl, Gordon Forbes, Ewan Carr

Published 2026-03-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Oyebayo Ridwan Olaniran, Diana Shamsutdinova, Sarah Markham, Felix Zimmer, Daniel Stahl, Gordon Forbes, Ewan Carr

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to create the perfect new recipe for a soup that will be served to thousands of people. Before you open a restaurant, you need to know one crucial thing: How many test batches do you need to cook to be sure the soup tastes good?

If you cook too few batches, you might get lucky once, but the recipe could be a disaster for the next 1,000 customers. If you cook too many, you waste time, money, and ingredients.

This is exactly the problem doctors and researchers face when building Clinical Prediction Models (computer programs that predict if a patient will get sick, recover, or how long they might live). They need to know exactly how many patient records (the "test batches") they need to train their model so it works reliably in the real world.

This paper introduces a new tool called pmsims (a piece of software for the R programming language) that solves this problem much better than the old methods. Here is how it works, explained simply:

The Old Way: Guessing and Checking

Previously, researchers had two main ways to figure out the sample size:

The "Textbook Formula" (Analytical): This is like using a rigid recipe card that says, "If you have 10 ingredients, you need exactly 50 test batches."
- The Problem: Real life is messy. Patients aren't perfect ingredients; they have complex histories and weird correlations. The textbook formula often assumes everything is neat and tidy. When reality gets complicated, this formula gives the wrong answer, leading to models that fail later.
The "Brute Force" Simulation: This is like cooking 100 batches, tasting them, then cooking 101, tasting them, then 102... all the way up to 1,000.
- The Problem: It works, but it takes forever. It's like searching for a needle in a haystack by checking every single straw one by one. It's accurate but incredibly slow and expensive.

The New Way: The "Smart GPS" (Gaussian Process Search)

The pmsims package introduces a Gaussian Process (GP) search engine. Think of this as a Smart GPS for finding the right sample size.

Instead of checking every single number (1, 2, 3...) or relying on a rigid formula, the Smart GPS does something clever:

It Draws a Map: It cooks a few test batches (say, at 100, 500, and 1,000 patients) and sees how the model performs.
It Predicts the Terrain: Using a mathematical "surrogate model" (a fancy way of saying a smart guesser), it draws a map of what the performance would look like between those points. It knows that if the soup tastes great at 500 and terrible at 100, the "sweet spot" is probably somewhere in between.
It Zooms In: Instead of checking every number, the GPS says, "I'm 90% sure the answer is between 400 and 600. Let's only check those." It ignores the areas where it's already sure the answer isn't.
It Adapts: If the map looks foggy (uncertain), it cooks a few more batches in that specific area to clear the fog. If the map is clear, it moves on.

Why is this a Big Deal?

The researchers tested this new "Smart GPS" against the old "Brute Force" method and the "Textbook Formula" in thousands of different scenarios (predicting heart attacks, recovery times, etc.).

It's Faster: Because it skips the numbers it doesn't need to check, it finds the answer using fewer computer calculations. It's like finding the needle in the haystack by using a magnet instead of pulling out every straw.
It's More Stable: The old "Brute Force" method sometimes gave wildly different answers depending on how the computer generated the random data. The new GPS method gave consistent, reliable answers every time.
It Handles Complexity: Whether the data is simple or a chaotic mess of complex patient history, the GPS adapts. The old formulas broke down when things got complicated.

The "Assurance" Feature

The paper also highlights a feature called "Assurance."

The Mean Approach: "On average, this model will work." (Like saying, "If you flip a coin 100 times, you'll get 50 heads.")
The Assurance Approach: "We are 80% sure this model will work, even if we get a bit of bad luck." (Like saying, "We are 80% sure you'll get at least 45 heads.")

The pmsims tool can calculate the sample size needed to hit that high-confidence "Assurance" goal, which is much safer for patient care.

The Bottom Line

The authors found that for the best results, you should let the software run about 1,000 simulations (a very manageable number for modern computers) and check the results about 20 times for each step.

In short: The pmsims package is a smart, efficient, and flexible tool that helps researchers stop guessing and start knowing exactly how much data they need to build life-saving medical prediction tools. It replaces the slow, blind search with a smart, guided tour.

1. Problem Statement

Developing reliable and generalizable clinical prediction models requires determining an adequate sample size. Existing methods for sample size estimation suffer from two primary limitations:

Analytical Approaches (e.g., pmsampsize): These rely on closed-form formulas assuming linear predictors, normally distributed covariates, and maximum likelihood estimation. They often fail in real-world scenarios involving non-linearities, complex correlations, or machine learning models. Furthermore, they typically target a "mean" performance criterion, ignoring the variability inherent in finite training samples, which can lead to underpowered studies.
Simulation-Based Approaches (e.g., samplesizedev): While more flexible, traditional simulation methods often use exhaustive grid searches or deterministic bisection algorithms. These are computationally expensive, inefficient, and struggle to provide stable estimates (high variance) when the signal is weak or the dimensionality is high.

There is a lack of practical, flexible tools that can perform assurance-based sample size planning (ensuring a high probability of meeting performance targets) across diverse model types without prohibitive computational costs.

2. Methodology

The authors introduce and validate pmsims, an R package designed to estimate minimum sample sizes using a Gaussian Process (GP) surrogate modeling framework.

Core Workflow

The package integrates three components:

Data Generation: Simulates binary, continuous, and survival outcomes based on user-specified parameters (prevalence, event rate, signal strength, predictor dimensionality).
Model Fitting: Fits standard models (Logistic, Linear, Cox regression) to generated training data.
Performance Evaluation: Assesses models on an independent validation set using metrics like AUC (binary), $R^2$ (continuous), C-index (survival), and calibration slope.

Search Engines Compared

The study evaluates three search strategies to find the minimum sample size $n^*$ where expected performance $\geq$ target $\tau$ :

GP-Based Adaptive Search (gp): The primary innovation. It treats the sample size–performance relationship as an unknown smooth function. It uses a Gaussian Process surrogate to approximate this function, iteratively selecting the next sample size to evaluate based on an acquisition function (balancing exploration and exploitation). This drastically reduces the number of required model fits compared to grid searches.
Deterministic Bisection (bisection): A classical algorithm that halves the search interval based on performance at the midpoint.
Hybrid GP-Bisection (gp-bs): A three-stage approach using a coarse bisection search to narrow the interval, followed by a refined GP search.

Experimental Design

Aim 1 (Internal Validation): Compared the three search engines across binary, continuous, and survival outcomes. Variables included predictor counts ( $p=10, 100$ ), outcome prevalence/event rates, target metrics (discrimination vs. calibration), and simulation budgets ( $B=200$ to $2000$). Stability was measured via the Coefficient of Variation (CV) of the estimated $n^*$ over 100 replications.
Aim 2 (External Benchmarking): Compared the best-performing pmsims engine against pmsampsize (analytical) and samplesizedev (simulation-based). Validated by training models on the recommended sample sizes and testing them on independent datasets of 30,000 observations to measure achieved performance deviation from the target.

3. Key Contributions

Development of pmsims: A flexible R package that decouples sample size planning from specific analytical assumptions, allowing researchers to define custom data-generating processes and model classes.
Adaptive GP Surrogate Modeling: Demonstrates that Bayesian optimization principles (via GP) can efficiently navigate the sample size search space, significantly outperforming deterministic methods in stochastic environments.
Assurance-Based Planning: Provides a robust framework for "assurance" criteria (e.g., 80% probability of meeting a target), addressing the risk of underperformance that "mean-based" criteria miss.
Comprehensive Validation: Extensive simulation studies covering binary, continuous, and time-to-event outcomes, validating the method against both analytical and existing simulation-based tools.

4. Key Results

Aim 1: Search Engine Performance

Stability: The GP-based engine (gp) consistently yielded the most stable sample size estimates (lowest CV) across all outcome types and scenarios. It ranked highest in 9 out of 12 configurations.
Efficiency: The gp engine achieved a plateau in precision (CV) at a total simulation budget of $B \approx 1000$ with $\kappa = 20$ replications per evaluation. Beyond this point, gains were minimal.
High-Dimensional/Low-Signal Settings: The advantage of the GP approach was most pronounced in challenging scenarios (e.g., low event prevalence, high predictor dimensionality $p=100$ ), where deterministic bisection failed to converge stably.
Ranking: The gp engine achieved a perfect average rank of 1.00 in 9/12 configurations. The deterministic bisection method consistently ranked last due to inefficiency in stochastic settings.

Aim 2: Benchmark Comparison

vs. pmsampsize (Analytical): pmsampsize frequently underestimated required sample sizes in high-discrimination settings (deviations up to -9.84%), leading to models that failed to meet performance targets.
vs. samplesizedev (Simulation): pmsims (using the gp engine) achieved performance deviations within $\pm 1\%$ of the prespecified target across all outcome types, comparable to samplesizedev but with greater stability and flexibility.
Conservatism: The assurance-based criterion in pmsims recommended larger sample sizes than the mean-based criterion, resulting in consistently minor negative deviations (i.e., slightly exceeding the target), ensuring robustness.

5. Significance and Conclusion

The paper establishes pmsims as a superior tool for sample size planning in clinical prediction modeling.

Practical Impact: It offers a computationally efficient alternative to exhaustive simulations and a more robust alternative to restrictive analytical formulas.
Reliability: By using adaptive GP search, researchers can achieve principled sample size estimates with fewer computational resources, ensuring that developed models are less likely to be overfitted or underpowered.
Recommendations: The authors recommend using the GP-based engine with $\kappa \geq 20$ replications per evaluation and a budget of $B \approx 1000$ . This configuration provides the optimal trade-off between estimation stability and computational cost.

The study concludes that adaptive surrogate-based search represents a significant methodological advance, enabling researchers to tailor sample size calculations to specific, complex clinical contexts rather than forcing their problems to fit available software assumptions.