📊 epidemiology

BUDS: Benchmark Uncertainty Design Selection for Two-Stage Single-Arm Phase II Trials

The paper introduces BUDS, a framework for selecting two-stage single-arm Phase II trial designs that accounts for historical benchmark uncertainty by optimizing efficiency across a plausible range of outcomes, thereby preventing Type I error inflation while offering explicit trade-offs between robustness and sample size efficiency for both binary and time-to-event endpoints.

Original authors: Irlmeier, R., Jin, Z., Ye, F.

Published 2026-07-29

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Irlmeier, R., Jin, Z., Ye, F.

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine you are a detective trying to catch a thief, but you don't know exactly what the thief looks like. You have a sketch, but it's a bit fuzzy. In the world of medical science, specifically in the early stages of testing new drugs, researchers play a similar game. They want to know if a new treatment works better than the "old way" of doing things. To do this, they run what are called "Phase II trials." Think of these as the first serious audition for a new actor. The researchers compare the new drug's performance against a "historical benchmark"—a standard number based on how patients usually do with the current best treatment. It's like saying, "If the new drug cures more than 10% of people, we'll keep going; if not, we stop."

The problem is, that "10%" number is often a guess. It might come from a small study, or the patients might be slightly different, or medicine might have changed a little since the old study was done. If the researchers pick the wrong number for their benchmark, they might accidentally think a useless drug is a hero, or they might throw away a good one too soon. This paper tackles the tricky math of how to design these drug auditions when you aren't 100% sure what the "standard" performance actually is. It's about building a safety net so that even if your guess is a little off, you don't make a costly mistake.

The "BUDS" Solution: Designing for the Unknown

This paper introduces a new framework called BUDS (Benchmark Uncertainty Design Selection). Think of BUDS as a smart, cautious architect designing a bridge. Usually, when you build a bridge, you calculate the load based on a single, specific weight—say, a car that weighs exactly 2,000 pounds. You design the bridge to hold that car perfectly. But what if, in reality, the cars crossing the bridge could weigh anywhere between 1,800 and 2,200 pounds? If you only designed for 2,000, a heavy truck might snap your bridge.

The authors argue that traditional drug trial designs are like that single-weight bridge. They pick one specific "benchmark" (like a 10% cure rate) and design the whole study around it. The paper shows that if the real-world benchmark is actually higher (say, 15%), these traditional designs can fail spectacularly. They might declare a drug a success when it's actually a dud. In their simulations, the authors found that when the true benchmark was just a bit higher than planned, the error rate for a popular design called "Simon Optimal" skyrocketed to 0.407 (that's a 40.7% chance of a false alarm!) instead of the intended 10%.

To fix this, BUDS suggests stopping the "single guess" game. Instead of picking one number, researchers should define a plausible range of what the benchmark could be (e.g., "The cure rate is likely between 5% and 15%"). BUDS then designs the trial to work safely across that entire range.

How BUDS Works: The "Worst-Case" and "Average" Strategies

The paper proposes two main ways to pick the best trial design from the many options that fit this new, wider safety net:

BUDS-Worst-Regret: This is the "paranoid" strategy. It asks, "What is the worst-case scenario within our range, and how much extra work (patients) would we need to handle that specific worst case?" It then picks the design that minimizes the maximum disappointment or "regret" if the worst case turns out to be true. It's like packing a suitcase for a trip where you might need to carry a heavy coat, a light jacket, or nothing at all. You choose the outfit that keeps you comfortable even if you end up in the coldest weather, even if it means you're slightly overdressed for a sunny day.
BUDS-Avg-EN: This is the "average" strategy. It looks at the whole range of possibilities and picks the design that uses the fewest patients on average. It's like packing for a trip where you expect a mix of weather and just want to be efficient overall, even if you might get a little chilly on one specific day.

What the Simulations Showed

The authors ran thousands of computer simulations to see how these new designs compare to the old ones. Here is what they found:

Safety First: The most important finding is that BUDS designs keep the "false alarm" rate (Type I error) under control no matter where the true benchmark falls within the range. Whether the benchmark is at the low end or the high end of the guess, the trial won't accidentally declare a bad drug a winner. In contrast, the old single-benchmark designs saw their error rates explode when the real benchmark was higher than expected.
The Cost of Safety: There is a trade-off. To be safe across a wide range of possibilities, BUDS designs sometimes require more patients than the old designs. For example, in a scenario where the benchmark was uncertain, a traditional design might need a maximum of 37 patients. The BUDS design, to be safe against the worst-case scenario, might need 54 or even 72 patients. The paper notes that this extra enrollment is the price of not making a mistake. It's better to test a few extra people than to waste millions on a Phase III trial for a drug that doesn't work.
Real-World Examples: The authors tested their ideas on real-world scenarios, like a trial for glioblastoma (a type of brain cancer). They looked at a trial that used a benchmark of 0.10 (10%). When they applied BUDS with a realistic range of 0.05 to 0.13, the old design would have had a false alarm rate of over 0.25 (25%). The BUDS design kept the error rate below 0.093 (9.3%), but it required a maximum sample size of 89 patients instead of the original 50.

The Bottom Line

The paper concludes that BUDS offers a transparent way to handle the uncertainty of medical history. It doesn't magically make the uncertainty disappear; instead, it forces researchers to admit, "We aren't 100% sure, so let's plan for a range." By doing so, it makes the trade-off between efficiency (using fewer patients) and robustness (avoiding false positives) explicit. The authors have even built a free computer tool (an R package and a Shiny app) so other scientists can use this method to design their own trials.

In short, BUDS suggests that in the high-stakes game of drug testing, it's smarter to design a bridge that can handle a range of weights than to gamble on a single, perfect guess. The paper doesn't claim this is the final word on all medical trials, but it provides a solid, mathematically sound way to stop guessing and start planning for the unknown.

Technical Summary: Benchmark Uncertainty Design Selection (BUDS) for Two-Stage Single-Arm Phase II Trials

Problem Statement
Single-arm Phase II clinical trials, particularly in oncology, frequently rely on two-stage designs (e.g., Simon's design for binary endpoints or the Kwak and Jung/r-KJ framework for time-to-event endpoints) to screen experimental therapies against a historical control benchmark. These designs are traditionally calibrated using a single, fixed planning benchmark ( $\theta_0$ ). However, in practice, historical benchmarks are often uncertain due to small sample sizes in prior studies, population heterogeneity, evolving eligibility criteria, and changes in standard of care.

The paper identifies a critical vulnerability in current practice: when the true historical benchmark exceeds the value used to calibrate the design, the Type I error rate can be substantially inflated. This inflation increases the risk of advancing ineffective treatments to costly Phase III trials. Existing design-selection criteria typically optimize efficiency (e.g., minimizing expected sample size) at a single benchmark point without accounting for this uncertainty, leaving trials statistically valid only at the specific calibration point but potentially invalid across the range of plausible historical values.

Methodology
The authors propose Benchmark Uncertainty Design Selection (BUDS), a framework that explicitly incorporates a prespecified, clinically plausible range of benchmark values ( $\theta_{0L}, \theta_{0U}$ ) into the design-selection process.

Framework Definition: Instead of selecting a design based on a single $\theta_0$ , BUDS requires that candidate designs satisfy Type I error ( $\alpha$ ) and Type II error ( $\beta$ ) constraints across the entire benchmark range. Due to the monotonicity of Type I error rates with respect to the benchmark parameter, the authors demonstrate that enforcing the Type I error constraint at the upper bound of the range ( $\theta_{0U}$ ) is sufficient to control error across the entire interval.
Objective Functions: Within the set of feasible designs that satisfy error constraints over the range, BUDS introduces two robust optimization objectives:
1. BUDS-Worst-Regret: Minimizes the maximum "regret" across the benchmark range. Regret is defined as the difference between the expected sample size (ESS) of a candidate design and the ESS of the locally optimal design (the design that would be chosen if the true benchmark were known) at any specific point in the range. This approach hedges against the worst-case efficiency loss.
2. BUDS-Avg-EN: Minimizes the average expected sample size across the benchmark range, favoring designs that are efficient on average.
Algorithm: The authors provide computationally efficient search algorithms to identify optimal designs. For binary endpoints, the method exhaustively enumerates candidate two-stage configurations. For time-to-event (TTE) endpoints, it utilizes a "zoom" search procedure based on the restricted-Kwak and Jung (r-KJ) framework. The implementation is available in an open-source R package (BUDS) and an interactive Shiny application.

Key Results
The paper evaluates BUDS against classic single-benchmark designs (Simon Optimal/Minimax and r-KJ) across representative binary and TTE scenarios with varying degrees of benchmark uncertainty.

Robustness to Misspecification: Under substantial benchmark uncertainty, single-benchmark designs exhibit severe Type I error inflation. For example, a Simon Optimal design calibrated at a response rate of 0.10 (with $\alpha=0.10$ ) reached a Type I error rate of 0.407 when the true benchmark was 0.15. Similarly, an r-KJ design calibrated at a 6-month survival probability of 0.50 (with $\alpha=0.05$ ) saw its error rate inflate to approximately 0.17 when the true survival was 0.67. In contrast, BUDS-selected designs maintained Type I error rates at or below the nominal level across the entire benchmark range.
Efficiency Trade-offs: Achieving robustness requires additional information. As the benchmark range widens, BUDS-selected designs require larger maximum sample sizes compared to designs calibrated at a single point. However, the paper notes that BUDS-Worst-Regret designs often achieve lower average expected sample sizes across the range than the "Minimax" single-benchmark designs and smaller first-stage sample sizes than both Simon Optimal and Minimax designs.
Real-World Application: Applying BUDS to recurrent glioblastoma trial settings (both binary response and 6-month progression-free survival) demonstrated that even modest uncertainty ranges (e.g., 0.05 to 0.13 for response rates) necessitate design adjustments. BUDS-selected designs controlled Type I error across these ranges, whereas classic designs failed to do so, often requiring only a modest increase in sample size (e.g., 6 additional patients in one TTE scenario) to achieve robustness.

Significance and Claims
The paper positions BUDS as a transparent framework for trial planning that makes the trade-off between robustness and efficiency explicit. Its primary significance lies in:

Error Control: It ensures Type I error control across a plausible range of historical benchmarks rather than at a single, potentially unrepresentative point, thereby reducing the risk of advancing ineffective therapies due to benchmark misspecification.
Decision Transparency: By requiring the specification of a benchmark range, the framework forces a structured discussion of uncertainty during protocol development, supporting more defensible go/no-go decisions.
Practical Utility: The framework applies to both binary and time-to-event endpoints while retaining the operational simplicity of classic two-stage designs. The availability of the BUDS R package and Shiny app facilitates immediate adoption by researchers.

The authors conclude that while BUDS does not eliminate the inherent limitations of single-arm studies, it provides a rational strategy to mitigate the downstream risks of false positives in Phase III trials by treating robustness as an explicit planning criterion rather than a post hoc consideration.

The "BUDS" Solution: Designing for the Unknown

How BUDS Works: The "Worst-Case" and "Average" Strategies

What the Simulations Showed

The Bottom Line

More like this