📊 statistics

Evaluating the impact of outcome delay on the efficiency of sample size re-estimation

This paper evaluates how outcome delays in clinical trials with sample size re-estimation reduce efficiency by inflating the average sample size and power, particularly when the re-estimated sample size is smaller than originally planned, leading to costly overpowered trials.

Original authors: Aritra Mukherjee, Michael J Grayling, James J M S Wason

Published 2026-05-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Aritra Mukherjee, Michael J Grayling, James J M S Wason

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are organizing a massive party to test a new recipe. You need to invite enough people to be sure the recipe is a hit, but you don't want to invite too many, or you'll waste money on food that nobody eats.

In the world of clinical trials (testing new medicines), scientists face this exact problem. They have to guess how many patients to enroll to prove a drug works. This guess depends on a "nuisance parameter"—a fancy way of saying "a variable we aren't sure about," like how much the results might vary from person to person.

The Solution: The "Internal Pilot" (The Taste Test)
To fix their guess, researchers use a strategy called Sample Size Re-estimation (SSR). Think of this as a "taste test" halfway through the party.

They invite a small group of people (the first stage).
They check the data to see how variable the results are.
Based on that, they adjust the total number of guests they need. If the results are very consistent, they might need fewer people. If they are all over the place, they might need more.

The Problem: The "Pipeline" (The Slow Cook)
Here is where things get tricky. In many medical trials, the "result" (did the patient get better?) doesn't happen immediately. It might take months to see if the drug worked. This is called Outcome Delay.

Imagine you are the party host. You've invited 60 people for the taste test. You are waiting for the results of their feedback before you decide how many more people to invite.

The Ideal Scenario: You stop inviting new people while you wait for the feedback.
The Real Scenario: You keep inviting people because the invitation process is automatic. While you are waiting for the 60th person's feedback (which takes 8 months), you keep inviting new guests.

By the time the feedback arrives, you might have invited 100 people total. But the taste test told you that you only needed 80 people to get a good answer. You now have 20 "pipeline" guests—people who are at the party but whose feedback won't count toward your decision. You've wasted resources on an over-powered trial (too many people, too much cost).

What This Paper Found
The authors of this paper ran thousands of computer simulations to see how bad this "waiting game" actually is. They used three main ways to measure the damage:

The "Delay Impact" (How often do we overshoot?):
- They found that the longer you wait for the results, the more likely you are to accidentally invite too many people.
- The Big Surprise: This problem is worst when the drug works better than expected (or the data is less variable than you thought). In these cases, the "taste test" tells you to cut the guest list down. But because you kept inviting people while waiting, you end up with a huge crowd anyway.
- Conversely, if the data is messy and you need more people, the delay isn't as bad. The extra people you invited while waiting actually help you reach the higher number you needed.
The "RMSE" (How far off are we?):
- This measures how far the final number of guests is from the "perfect" number.
- The study shows that as the delay gets longer, the final number of guests drifts further away from the ideal. The trial becomes less efficient.
The "Cost" (The Penalty Score):
- The authors created a special score that punishes "under-powered" trials (not enough people to prove the drug works) more severely than "over-powered" ones.
- They found that while delays usually lead to over-powering (wasting money), the specific "cost" of the delay depends heavily on the initial guess. If you guessed the variance was high (needed many people) but it turned out to be low (needed few), the delay causes a massive spike in cost because you recruited way more than necessary.

The Recipe for Success
The paper concludes with a specific recommendation on how to handle this "waiting period":

Don't wait too long to check: The timing of your "taste test" matters.
The Sweet Spot: The authors suggest that for continuous data (like blood pressure readings), checking the data after 35 people per group is the best balance.
- If you check too early (fewer people), your guess might be wild and inaccurate.
- If you check too late (many people), you risk inviting a huge "pipeline" of unnecessary guests while you wait for the results.

In a Nutshell
Delaying the results of a clinical trial while you keep recruiting patients is like cooking a stew and keeping the pot full of water while you wait for the soup to boil. If you need less water than you thought, you've wasted a lot of it. This paper shows that this waste is most dangerous when the trial turns out to be "easier" than expected. To avoid this, researchers should do their "taste test" (re-estimation) after about 35 patients per group, rather than waiting until they have recruited a huge crowd.

Technical Summary: Evaluating the Impact of Outcome Delay on the Efficiency of Sample Size Re-estimation

Problem Statement
Sample size re-estimation (SSR) is a widely used adaptive design intended to ensure clinical trials meet pre-specified power requirements when nuisance parameters (e.g., variance, event rates) are uncertain at the planning stage. While SSR designs, particularly blinded "internal pilot" types, offer flexibility, they typically assume that treatment outcomes are available immediately upon participant recruitment. In practice, many trials involve long-term primary endpoints where outcomes are delayed.

This delay creates a "pipeline" effect: if recruitment continues while waiting for interim outcome data, a significant number of participants may be recruited whose outcomes are not yet available for the interim analysis. If the delay is substantial, the trial may recruit more participants than the re-estimated sample size deems necessary. This leads to an over-powered trial, incurring unnecessary financial costs and resource waste. Conversely, pausing recruitment to await outcomes prolongs trial duration. Previous literature has established that outcome delays negatively impact group-sequential and multi-stage designs, but the specific efficiency losses for SSR designs, particularly regarding the distribution of the final sample size and the precision of the re-estimation, required further investigation.

Methodology
The authors investigate the impact of outcome delay on the efficiency of blinded SSR designs using a simulation study. The study focuses on continuous outcomes (normal distribution) and extends to binary outcomes in supplementary materials.

Design Framework: The study assumes a two-stage design where an interim analysis is conducted after $n_1$ participants are recruited. The nuisance parameter (variance $\sigma^2$ ) is re-estimated using the pooled sample variance of the blinded data. The sample size is adjusted based on this re-estimation to achieve a target power.
Delay Modeling: The study models a delay period $m$ $m$ (time units) required to observe primary outcomes. Recruitment is assumed to continue during this period. The number of "pipeline" participants ( $n_{delay}$ $n_{d e l a y}$ ) recruited during the delay is calculated under two recruitment patterns:
1. Uniform: Constant recruitment rate (Poisson arrival).
2. Linear: Recruitment rate increases linearly over time (simulating new sites opening).
Final Sample Size ( $N^*$ ): The final sample size is determined by the re-estimated requirement ( $n^*$ ) and the pipeline participants. If $n^*$ is greater than the pipeline accumulation, $N^* = n^*$ . If the pipeline accumulation exceeds the re-estimated need, $N^* = n_1 + n_{delay}$ , resulting in an over-recruited trial.
Scenarios: Simulations were run for 10,000 replicates across three variance scenarios relative to the initial assumption ( $\sigma^2_{init}$ $σ_{ini t}^{2}$ ):
- Case I: True variance is lower ( $\sigma^2 = 8 < 10$ ), leading to a re-estimated sample size smaller than planned.
- Case II: True variance matches assumption ( $\sigma^2 = 10 = 10$ ).
- Case III: True variance is higher ( $\sigma^2 = 12 > 10$ ), leading to a re-estimated sample size larger than planned.
Efficiency Metrics: The authors propose and utilize three metrics to quantify the impact of delay:
1. Delay Impact: The proportion of trials where the final sample size exceeds the re-estimated requirement due to pipeline participants.
2. Root Mean Square Error (RMSE): Measures the precision of the final sample size relative to the "oracle" sample size (the size required if the true variance were known from the start).
3. Cost: A novel metric defined as $E[(N^* - n_{oracle})^2 / 100 \times Power(N^*)]$ . This penalizes under-powered trials more severely than over-powered ones, reflecting the higher consequence of failing to detect a treatment effect compared to the cost of excess recruitment.

Key Results
The simulation results indicate that the impact of outcome delay is highly dependent on the relationship between the true variance and the initially planned sample size, as well as the recruitment pattern.

Distribution of Final Sample Size: As the delay length ( $m$ ) increases, the spread of the final sample size distribution decreases because the minimum attainable sample size rises (due to the accumulation of pipeline participants).
Case I (Over-planned/True Variance Lower): This scenario suffers the most from delay. Since the re-estimated sample size is smaller than the initial plan, the pipeline participants often push the total recruited count above the required $n^*$ . This leads to a high "Delay Impact" (e.g., ~80% for a 10-month delay under uniform recruitment) and results in frequently over-powered trials. The RMSE and Cost metrics increase sharply with delay.
Case III (Under-planned/True Variance Higher): This scenario is least impacted by delay. The re-estimated sample size is larger than the initial plan, so pipeline participants contribute positively toward the required total. The delay impact is minimal, and in some cases (uniform recruitment), continuing recruitment during the delay may even improve efficiency compared to pausing recruitment.
Recruitment Patterns: Linear recruitment patterns generally result in higher costs and greater delay impacts compared to uniform patterns because the accelerating recruitment rate generates more pipeline participants during the delay period.
First-Stage Sample Size ( $n_1$ ): Increasing $n_1$ reduces the variability of the re-estimated sample size but increases the minimum final sample size in the presence of delay. The authors find that a first-stage sample size of 35 participants per arm (total 70) strikes a balance between obtaining a reliable variance estimate and limiting the severity of delay-induced over-recruitment.

Significance and Claims
The paper claims that while SSR is a powerful tool, ignoring outcome delay in the planning stage can lead to significant efficiency losses, primarily manifesting as over-powered trials with inflated costs. The study demonstrates that:

Context-Dependent Impact: The severity of the delay is not uniform; it is most detrimental when the true variance is lower than anticipated, causing the trial to require fewer participants than originally planned.
Metric Innovation: The introduction of the "Cost" metric and "Delay Impact" provides a more nuanced understanding of efficiency loss than traditional metrics, specifically highlighting the risk of over-recruitment.
Practical Guidance: The results suggest that for continuous outcome data, conducting the interim analysis after recruiting 35 participants per arm is a reasonable strategy. This aligns with previous external pilot recommendations (Teare et al.) but extends them to the context of blinded internal pilot SSR, balancing the need for variance estimation precision against the risk of pipeline accumulation.
Relative Robustness: The authors note that while delay impacts SSR, the effect is less severe than in designs allowing early stopping for efficacy or futility (e.g., Simon's two-stage or group-sequential designs).

The authors conclude that careful simulation studies accounting for specific recruitment patterns and delay lengths are essential for planning SSR trials to mitigate potential financial and resource disadvantages. They do not propose new algorithms for handling delay but rather provide a quantitative assessment of the existing "blind" approach under delayed conditions.

Technical Summary: Evaluating the Impact of Outcome Delay on the Efficiency of Sample Size Re-estimation

More like this