📊 statistics

Considerations for the Integration of Randomized Controlled Trials and Real-World Data

This article outlines a principled framework for integrating randomized controlled trials and real-world data to support individualized clinical decision-making, detailing how distinct objectives shape design and analytic choices while addressing practical challenges like data curation, comparability, and sensitivity analysis to maintain regulatory-grade evidentiary standards.

Original authors: Sky Qiu, Charles Barr, Lauren Dang, Issa Dahabreh, Larry Han, Kajsa Kvist, Hana Lee, Andrew Mertens, Nerissa Nance, Lei Nie, Kara Rudolph, Xu Shi, Jens Tarp, Salina P. Waddy, Kenneth Wiley, Andy Wilso

Published 2026-04-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Sky Qiu, Charles Barr, Lauren Dang, Issa Dahabreh, Larry Han, Kajsa Kvist, Hana Lee, Andrew Mertens, Nerissa Nance, Lei Nie, Kara Rudolph, Xu Shi, Jens Tarp, Salina P. Waddy, Kenneth Wiley, Andy Wilson, Margot Lisa Jing Yann, Zhiwei Zhang, Tianyue Zhou, Maya Petersen, Mark van der Laan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to perfect a new recipe for a soup that will feed a whole city.

The Gold Standard: The Controlled Kitchen (RCT)
Traditionally, to prove your soup is delicious and safe, you run a Randomized Controlled Trial (RCT). This is like a high-end, sterile test kitchen. You pick a very specific group of people (say, only left-handed chefs aged 30-40), give them strict instructions, control the temperature of the stove, and measure every ingredient precisely.

The Good: Because you control everything, you know exactly that the soup tastes good. The result is scientifically "true" for that specific group.
The Bad: The test kitchen is small, expensive, and the people in it aren't like the average person in the city. Maybe the city has many elderly people, or people who cook in small apartments with weak stoves. Your "perfect" soup might taste different (or even be unsafe) when 10,000 regular people try to make it in their own messy kitchens.

The Real World: The Street Food Festival (RWD)
Then there is Real-World Data (RWD). This is like watching thousands of people make your soup at a chaotic street food festival. They use different pots, forget ingredients, turn the heat up and down, and have all sorts of dietary restrictions.

The Good: It shows you how the soup actually performs in the real world. It covers a huge variety of people and long-term effects.
The Bad: It's messy. You don't know if they liked the soup because of your recipe, or because they added extra salt, or because they were just hungry. It's hard to tell cause from effect.

The Big Idea: Mixing the Two
This paper is a guidebook for mixing the Test Kitchen with the Street Festival. The goal is to get the scientific rigor of the test kitchen plus the real-world relevance of the festival.

The authors (a team of statisticians, doctors, and regulators) say: "Don't just look at the test kitchen results and guess how they apply to the world. Don't just look at the festival and guess if the recipe is good. Integrate them."

Here is how they suggest doing it, using simple analogies:

1. Why Mix Them? (The Objectives)

The paper lists five reasons to mix the data, like adding different ingredients to a stew:

More Power (Bigger Sample): If your test kitchen only had 10 people, you might miss a rare side effect (like a stomach ache that happens to 1 in 1,000). The street festival has 10,000 people. Combining them helps you spot those rare events.
Generalizability (The "Average" Person): Your test kitchen only had 30-year-olds. The street festival has kids, seniors, and people with different health issues. By mixing them, you can say, "This soup works for everyone, not just 30-year-olds."
Transporting (The "Different" City): Maybe your test kitchen was in New York, but you want to know if the soup works in Tokyo (different climate, different ingredients). You use the festival data from Tokyo to "transport" your findings there.
Long-Term Watch (The "Time" Machine): Test kitchens usually only run for 6 months. The street festival has been going on for 10 years. You can link your test kitchen participants to the festival records to see what happens to them 5 or 10 years later.
When You Can't Have a Control Group: Sometimes, for rare diseases, you can't find enough people for a test kitchen. You might only have one group of patients. In this case, you use the "festival" data as a fake control group to compare against.

2. The Danger Zone: The "Apples and Oranges" Problem

The paper warns that you can't just dump the festival data into the test kitchen data. That's like trying to compare a Michelin-starred chef's knife skills with a toddler's butter knife. They aren't the same!

The "Misalignment" Checklist:
Before mixing, you must check for mismatches:

Time Mismatch: Did the festival happen 20 years ago when medicine was different? (Like comparing soup made with 1990s ingredients to 2024 ingredients).
Definition Mismatch: In the test kitchen, "eating the soup" means finishing the bowl. In the festival, does "eating" mean taking one spoonful? You need to make sure you are measuring the same thing.
Selection Bias: In the festival, maybe only the people who liked the soup stayed to fill out a survey. If you only look at them, you think the soup is perfect. You have to be careful not to pick only the "happy" customers.

3. The Recipe for Success (The Causal Roadmap)

The authors propose a step-by-step "Causal Roadmap" (a fancy term for a recipe) to ensure the mix works:

Ask the Right Question: Are you asking "Does it work in the test kitchen?" or "Does it work for the whole city?"
Define the Target: Be specific about who you are talking about.
Check the Assumptions: Be honest about what you are guessing. "We assume the festival people are similar enough to the test kitchen people if we account for their age and health."
Run Sensitivity Checks: This is like tasting the soup at different stages. "What if my assumption about age was wrong? Does the result change?" If the result flips completely, your recipe is shaky.

4. The "Magic" Tools (Estimators)

The paper mentions fancy statistical tools (like "Bayesian borrowing" or "Conformal prediction"). Think of these as smart filters.

Instead of blindly mixing all the festival data, these tools act like a sieve. They say, "We will only borrow data from the festival customers who look very similar to our test kitchen customers."
This prevents the "noisy" festival data from ruining the "clean" test kitchen data.

The Bottom Line

This paper is a plea for honesty and precision.
It tells regulators and scientists: "We have amazing tools to mix the perfect test kitchen with the messy real world. But we must do it carefully. We can't just throw data together and hope for the best. We need to define our goals, check our ingredients, and admit where we are making guesses."

If done right, this integration gives us better medicine: treatments that are proven to work in the lab and proven to work for real people in real life.

1. Problem Statement

The pharmaceutical and regulatory landscape is shifting from relying solely on Randomized Controlled Trials (RCTs) toward integrating Real-World Data (RWD) to generate Real-World Evidence (RWE). While RCTs remain the gold standard for internal validity, they often suffer from limitations regarding:

Statistical Power: Inability to detect effects on rare subgroups or secondary endpoints.
Generalizability: Trial populations often differ significantly from the broader target population in clinical practice.
Follow-up Duration: RCTs are typically short-term, limiting the assessment of long-term safety and outcomes.
Feasibility: In rare diseases or specific oncology settings, concurrent control arms may be unethical or impossible, necessitating external controls.

The core challenge is that naive integration of RCT and RWD introduces biases due to misalignments between data sources, including differences in patient eligibility, treatment implementation, outcome definitions, coding systems, and calendar time (non-concurrency). There is a lack of a unified, principled framework to guide stakeholders on how to design, analyze, and interpret these hybrid studies while maintaining regulatory-grade evidentiary standards.

2. Methodology and Framework

The paper presents a consensus statement developed by the Forum on the Integration of Observational and Randomized Data (FIORD). The methodology is grounded in the Causal Roadmap framework (Petersen and van der Laan, 2014), which provides a structured, iterative approach to causal inference.

Key methodological components include:

A. The Causal Roadmap Application

The authors apply the Causal Roadmap to RCT-RWD integration, emphasizing that the causal estimand (the specific quantity to be estimated) must be defined before selecting statistical methods. The roadmap involves:

Defining the causal question and target population.
Specifying the causal estimand.
Articulating identification assumptions.
Selecting appropriate statistical estimators.
Conducting sensitivity analyses.

B. Structural Causal Model (SCM) and Notation

The paper formalizes the data structure as $O = (W, S, A, Y)$ , where:

$W$ : Baseline covariates.
$S$ : Data source indicator ( $S=1$ for RCT, $S=0$ for RWD).
$A$ : Treatment assignment.
$Y$ : Outcome.

They define potential outcomes under hypothetical interventions on both treatment ( $A$ ) and study source ( $S$ ), allowing for the definition of estimands that account for protocol adherence and contextual effects specific to the trial setting.

C. Identification Assumptions

The paper details specific assumptions required for non-parametric identification of different estimands:

Standard RCT Assumptions: Consistency, Exchangeability (conditional on $W$ ), and Positivity of treatment within the RCT.
Integration-Specific Assumptions:
- Positivity of Trial Participation: Every patient stratum in the pooled population has a non-zero probability of being in the RCT.
- Transportability/Generalizability: Conditional on measured covariates $W$ , trial participation ( $S$ ) provides no additional information about potential outcomes (i.e., $W$ captures all effect modifiers and prognostic factors).

D. Estimators and Methods

The paper reviews several classes of estimators:

Test-then-Pool: Hypothesis testing for exchangeability before pooling data.
Bayesian/Dynamic Borrowing: Using power priors or commensurate priors to down-weight external data when conflicts arise.
Bias-Correction (e.g., Adaptive TMLE): Explicitly estimating and correcting for the bias introduced by the study indicator.
Selective Borrowing: Using conformal prediction to include only external patients with covariate profiles highly similar to the trial cohort.

3. Key Contributions

A. Taxonomy of Integration Objectives

The authors categorize RCT-RWD integration into five distinct objectives, each requiring different design considerations and estimands:

Improving Statistical Power: Augmenting RCTs to analyze rare events or subgroups (e.g., Asian subgroups in the DEVOTE trial).
Improving Generalizability: Reweighting the RCT sample to reflect the distribution of a broader, trial-eligible target population.
Transporting Findings: Estimating effects in populations different from the trial population (e.g., lower-risk patients), requiring stronger transportability assumptions.
Understanding Long-term Outcomes: Linking RCT participants to post-trial RWD to assess long-term safety and durability of effects.
External Controls: Constructing control arms from RWD for single-arm trials where internal controls are infeasible.

B. Formalization of Distinct Estimands

The paper clarifies the subtle but critical differences between three primary estimands:

$\Psi_1$ : Average Treatment Effect (ATE) within the RCT population only.
$\Psi_2$ : ATE within the pooled RCT and RWD population (marginalizing over the pooled covariate distribution).
$\Psi_3$ : ATE under a hypothetical intervention where the entire target population is enrolled in the RCT regime.
Key Insight: $\Psi_1$ and $\Psi_2$ differ if there is effect modification by covariates and the covariate distributions differ between sources. $\Psi_3$ requires the strong assumption that measured covariates capture all differences between trial participants and non-participants.

C. Practical Checklist for Misalignment (Table 1)

The authors provide a comprehensive checklist for assessing Data Relevance and Data Reliability, identifying specific sources of bias:

Target Population/Eligibility: Can trial criteria be emulated in RWD?
Calendar-Time Mismatch: Risks of non-concurrency (changes in standard of care).
Care Setting Differences: Monitoring intensity and adherence protocols.
Variable Definitions: Harmonization of units, time windows, and coding (e.g., adjudicated endpoints vs. claims codes).
Treatment Definition: Differences in adherence, switching, and "multiple versions" of treatment.
Outcome Definition: Differences in event ascertainment and follow-up precision.

D. Emphasis on Sensitivity Analysis

The paper argues that sensitivity analyses should not be an afterthought but a core component of the primary analysis plan. It advocates for:

Reporting RCT-only estimates alongside integrated estimates.
Using negative controls (treatments or outcomes) to detect unmeasured confounding.
Calculating "G-values" to quantify the minimum bias required to overturn study conclusions.

4. Results and Illustrative Examples

While the paper is a consensus statement rather than an empirical study, it uses the DEVOTE RCT (evaluating insulin degludec vs. glargine) as a running example to illustrate concepts:

Power: Demonstrated how external RWD could augment the underpowered analysis of individual MACE components (cardiovascular death, MI, stroke) within the DEVOTE trial.
Generalizability: Showed how reweighting could adjust for the underrepresentation of Asian patients in the trial.
Estimand Differences: Illustrated scenarios where $\Psi_1$ and $\Psi_2$ would be approximately equal (e.g., if effect modification is absent or covariate distributions match) versus scenarios where they diverge significantly.
Bias-Variance Trade-off: Highlighted that expanding the calendar window for RWD increases sample size (reducing variance) but may introduce bias due to temporal changes in clinical practice.

5. Significance and Impact

Regulatory Guidance: The paper aligns with and synthesizes recent FDA and ICH guidance (e.g., ICH E9(R1), FDA RWD frameworks) to provide a practical, actionable guide for sponsors and regulators.
Paradigm Shift: It moves the field from qualitative "totality of evidence" assessments to explicit quantitative synthesis with transparent assumptions.
Prevention of "Post Hoc" Integration: The authors strongly advise against treating data integration as a "savior" for underpowered trials. Instead, they advocate for prospective planning, including pre-specifying RWD sources, target populations, and estimands in the study protocol.
Standardization: By defining a common language (via the Causal Roadmap) and a checklist for bias assessment, the paper aims to improve the transparency, reproducibility, and regulatory acceptance of hybrid RCT-RWD studies.

In conclusion, this paper establishes a rigorous, causal-inference-based framework for integrating RCTs and RWD. It emphasizes that successful integration depends not just on statistical methods, but on the careful alignment of study objectives, estimand definitions, data quality, and identification assumptions.