📄 intensive care and critical care medicine

Accounting for uncertainty in the expected treatment effect substantially increases the sample size required for randomised trials: implications for the feasibility of clinical trials in anaesthesia and critical care

This study demonstrates that incorporating uncertainty regarding the expected treatment effect into sample size calculations for anaesthesia and critical care trials substantially increases the required participant numbers compared to conventional methods, thereby challenging the feasibility of many proposed randomized trials.

Original authors: Sidebotham, D., Barlow, J.

Published 2026-06-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sidebotham, D., Barlow, J.

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine you are planning a massive cooking competition to see if a new secret spice makes a soup taste better. To prove it works, you need to invite enough judges (participants) to be absolutely sure the result isn't just luck.

This paper is about how researchers in anesthesia and critical care are planning these "competitions" (clinical trials) and why they might be inviting far fewer judges than they think they need.

The Old Way: The "Perfect Guess" Trap

Traditionally, when scientists plan a trial, they make a fixed guess about how much better the new treatment will be.

The Scenario: They say, "We are 100% sure this new drug will cut the death rate by 10%."
The Calculation: Based on that fixed 10% guess, they calculate they need, say, 800 people to prove it works.
The Problem: In the real world, we rarely know the exact number. Maybe the drug only works 5% of the time, or maybe it doesn't work at all. By pretending they know the exact number, they are setting a trap. If their guess is too optimistic, the trial ends up with too few people, and they might miss a real effect (a "false negative") or, if they do find a result, it might be a fluke.

The New Way: The "Uncertainty Umbrella"

The authors suggest a smarter way called Assurance. Instead of pretending they know the exact effect, they admit, "We think it's a 10% improvement, but it could realistically be anywhere between a 20% improvement and a 2% worsening."

They use a tool called a Design Prior to map out this uncertainty. Think of this like an umbrella that covers all the possible outcomes, not just the one they hope for.

The Big Discovery: You Need a Much Bigger Tent

When the researchers ran the numbers using this "uncertainty umbrella," they found a startling result: To get the same level of confidence, you need way more people.

The Analogy: Imagine you are trying to hear a whisper in a noisy room.
- The Old Method assumes the room is perfectly quiet. You think you only need 100 people to hear the whisper.
- The New Method admits the room is actually noisy and the whisper might be fainter than expected. To hear that whisper clearly in the noise, you actually need 300 people.

The paper found that when researchers account for realistic uncertainty (a "Coefficient of Variation" or CV of 0.5, which they say is a realistic level of doubt), the number of people needed jumps by 1.5 to 3.5 times.

The "Ceiling" on Success

The paper also discovered a hard limit, like a glass ceiling.

If your uncertainty is too high (meaning you are very unsure if the treatment works at all), there is a point where no amount of extra people will ever guarantee a successful trial.
It's like trying to find a specific grain of sand on a beach. If you don't even know which beach it's on, adding more buckets of sand won't help you find it. The math shows that if your uncertainty is too great, the maximum chance of success is capped at a certain percentage, no matter how big the trial gets.

Why This Matters for Anesthesia and Critical Care

The authors point out that many large trials in these fields fail to show statistically significant results. They suggest this isn't just bad luck; it's because the trials were planned using the "Perfect Guess" method. They invited too few people because they didn't account for the fact that the treatment effect is uncertain.

The Bottom Line

The Claim: Accounting for the fact that we don't know the exact treatment effect forces us to recruit significantly more patients to be sure of the results.
The Recommendation: The authors suggest using a specific level of uncertainty (a CV of 0.5) as a standard "default" setting. This balances being realistic about what we don't know while still making it possible to run a trial.
The Result: If researchers do this, they will realize their trials need to be much bigger to be feasible, or they will realize that some very small effects are simply too hard to prove with current resources.

In short: Stop pretending you know the exact answer before you start. If you admit you're guessing, you'll realize you need a much bigger crowd to be sure of the truth.

Technical Summary: Accounting for Uncertainty in Expected Treatment Effects

Problem Statement
Multicentre randomised trials in anaesthesia and critical care frequently report low rates of statistically significant differences. The authors posit that this phenomenon is partly attributable to conventional sample size calculation methods, which assume a fixed treatment effect. These "conditional power" methods fail to account for genuine uncertainty regarding the expected effect size—a fundamental reality that often motivates the trial in the first place. When researchers overestimate the plausible effect size (a common occurrence), the achieved power of the study falls below the design power, increasing the risk of Type II errors and the "winner's curse" (where significant results overestimate the true effect). The paper addresses the need for methods that incorporate this uncertainty to provide more realistic sample size estimates.

Methodology
The study employed simulations in R to compare three approaches for estimating power and sample size in two-arm randomised trials with binary outcomes (expected control event rate of 50%):

Conventional (Conditional) Power: Assumes a fixed effect size.
Frequentist Assurance: Uses a design prior to represent uncertainty in the treatment effect ( $\beta_1$ ) and calculates power by averaging across plausible effect sizes via Monte Carlo integration.
Bayesian Assurance: Uses a design prior for both the control event rate ( $\beta_0$ ) and treatment effect ( $\beta_1$ ), simulating data and fitting Bayesian logistic regression models to determine the proportion of trials meeting a success criterion (posterior probability of benefit > 97.5%).

Key methodological features include:

Design Priors: Normal (Gaussian) distributions were specified for $\beta_1$ with a constant Coefficient of Variation (CV) ranging from 0.1 to 2.0. This standardised the degree of prior uncertainty across different effect sizes.
Effect Sizes: Four scenarios were tested with expected risk differences (RD) of -10%, -5%, -2%, and -1%.
Computational Efficiency: To make Bayesian assurance computationally tractable, the authors utilised and validated a normal approximation to the Bayesian posterior distribution, comparing it against Markov chain Monte Carlo (MCMC) sampling on a subset of simulations.
Theoretical Derivation: The authors derived a theoretical limit for achievable power ( $power_\infty$ ) as the sample size approaches infinity.

Key Results

Sample Size Inflation: Incorporating uncertainty via design priors substantially increases the required sample size compared to conventional methods. At a CV of 0.5 (representing realistic prior uncertainty), both frequentist and Bayesian assurance methods required sample sizes approximately 1.5 to 3.5 times larger than the conventional method to achieve 80% or 90% power.
Theoretical Power Limit: The study derived a theoretical ceiling for power defined by the equation $power_\infty = \Phi(1/CV)$ $p o w e r_{\infty} = Φ (1/ C V)$ , where $\Phi$ $Φ$ is the cumulative distribution function of the standard normal. This limit depends solely on the CV of the design prior and is independent of the effect size or inferential methodology.
- For a CV of 0.5, the maximum achievable power is ~98%.
- For a CV of 1.19, the maximum achievable power drops to 80%.
- For a CV of 2.0, the maximum achievable power is ~69%.
Methodological Agreement: Frequentist and Bayesian assurance methods produced similar power curves across all scenarios. Minor differences were attributed to sampling variability, the regularising effect of weakly informative analysis priors in the Bayesian approach, and the fact that the Bayesian method sampled uncertainty in the control event rate ( $\beta_0$ ) while the frequentist method treated it as fixed.
Validation: The normal approximation to the Bayesian posterior distribution matched MCMC results closely, reducing computation time from days/weeks to less than 8 hours on a standard laptop.

Significance and Claims
The paper claims that accounting for clinical uncertainty in the expected treatment effect is critical for the feasibility of randomised trials in anaesthesia and critical care. The authors argue that:

Feasibility Implications: The substantial increase in required sample sizes when uncertainty is accounted for suggests that many trials, particularly those targeting small treatment effects or those with high prior uncertainty (high CV), may be unfeasible or substantially underpowered if designed using conventional methods.
Explanation for Low Success Rates: The failure to account for genuine uncertainty may explain the low proportion of multicentre trials in these fields that report statistically significant differences.
Practical Recommendation: The authors suggest a CV of approximately 0.5 as a reasonable default for design priors. This value accommodates realistic uncertainty while still allowing for adequate power (e.g., 80-90%) to be achievable.
Methodological Contribution: The study provides a validated, computationally efficient normal approximation for Bayesian assurance power, making these methods more accessible for trial planning without the prohibitive cost of full MCMC sampling.

The paper concludes that while conventional methods assume a fixed effect, assurance methods offer a more realistic guide for sample size estimation, directly informing decisions regarding the feasibility of conducting specific clinical trials.