Differences in Performance of Bayesian Dynamic Borrowing and Synthetic Control Methods: A Case Study of Pediatric Atopic Dermatitis

Technical Summary: Differences in Performance of Bayesian Dynamic Borrowing and Synthetic Control Methods

Problem Statement
In clinical trial design, particularly for pediatric populations, rare diseases, or scenarios where the standard of care differs significantly from proof-of-concept treatments, recruiting and retaining participants for a prospective control arm presents substantial challenges. Two methodologies have emerged to address this: Bayesian Dynamic Borrowing (BDB) and Synthetic Control Methods (SCM). While both aim to augment or replace prospective control arms using historical data, their performance has not been directly compared. This gap exists due to fundamental differences in their application, the nature of their outputs (a prior distribution vs. a generated control arm), and the metrics used to evaluate their quality. This study seeks to bridge this gap by comparing the statistical power and Type 1 error rates of BDB (utilizing a Meta-Analytic Predictive prior) and SCM within the specific context of Pediatric Atopic Dermatitis.

Methodology
The study utilized a case study approach based on six historical Randomized Controlled Trials (RCTs) involving pediatric atopic dermatitis. The primary outcome of interest was the EASI-75 response rate (75% improvement in the Eczema Area and Severity Index).

• Data Source: Six historical RCTs were selected, providing sample sizes and responder counts for control groups. The participant age ranges spanned from 6 months to 18 years, and interventions included various treatments such as Dupilumab, Baricitinib, and Tralokinumab.
• Bayesian Dynamic Borrowing (BDB): The R library RBesT was employed to construct a Meta-Analytic Predictive (MAP) prior. This prior was informed by the historical control data. The methodology involved simulating hypothetical new data to combine with the prior, allowing the model to dynamically adjust the weight of the historical data based on the similarity between the historical prior and the observed trial data (downweighting if heterogeneity is high).
• Synthetic Control Methods (SCM): The R library Synthpop was used to generate a synthetic control arm. This involved simulating control-group outcomes based on the sample sizes and observed responder counts from the historical studies. The overall control response probability was modeled as the combined mean response rate and its distribution.
• Simulation and Metrics: Across 10,000 simulated replicates, both methods were evaluated using Fisher's exact tests and analytical power calculations. The primary performance metrics were statistical power and Type 1 error rate. For the SCM, a synthetic dataset was created for each replicate with participant counts equal to the mean historical sample size.

Key Results
The comparison yielded the following performance metrics:

• BDB: Produced a power of 0.580 and a Type 1 error rate of 0.026. The resulting MAP prior had a mean response rate of 0.20 with a 95% credible interval of 0.02–0.71.
• SCM: Produced a power of 0.641 and a Type 1 error rate of 0.027. The synthetic control had a mean response rate of 0.24 with a 95% confidence interval of 0.14–0.36.

The study noted that the SCM model demonstrated slightly higher power (a difference of 0.061) while maintaining a Type 1 error rate nearly identical to that of BDB. The 95% credible interval for the BDB prior was notably wider than the 95% confidence interval for the synthetic control, reflecting the greater uncertainty inherent in the Bayesian prior when incorporating heterogeneous historical studies.

A secondary analysis adjusted the SCM to match the total effective sample size of the BDB design (n=64). Under this constraint, the SCM achieved a power of 0.639 but a Type 1 error rate of 0.312, illustrating the expected increase in false-positive risk when sample sizes are reduced.

Contributions and Significance
The primary contribution of this work is the first direct empirical comparison of BDB and SCM performance metrics (power and Type 1 error) within a unified case study framework. The authors emphasize that while the methods produced comparable control response estimates (with a difference of approximately 5%) and overlapping uncertainty intervals, they serve distinct operational roles:

1. Application Context: BDB is designed to minimize the total number of participants needed by borrowing information, making it suitable when recruitment is difficult but allocation to a control arm is feasible. SCM is designed to generate a control arm entirely from historical data, making it preferable when allocating patients to a control group is unethical or practically impossible.
2. Methodological Differences: BDB is adaptive, adjusting its influence based on trial data similarity, whereas SCM follows a "one and done" approach where the control arm is generated and then analyzed statically.
3. Decision Framework: The paper concludes that neither method is universally superior. The choice between BDB and SCM should be driven by the specific constraints of the potential trial, specifically the ethical feasibility of control arm allocation and the difficulty of recruitment.

Limitations and Scope
The authors explicitly state that the comparison is limited to the control arm. The results cannot be extrapolated to scenarios where the intervention arm is augmented, as treatment effects may interact with trial-specific factors (e.g., population, dropout) differently than baseline rates. Furthermore, the SCM in this study utilized simulated datasets derived from historical summary statistics rather than real individual-level patient data, which may have resulted in narrower confidence intervals than would be observed in a real-world application using raw data. The study maintains a modest stance, noting that the "better" method is contingent on the specific needs of the trial rather than a definitive superiority in statistical metrics.