Treatment effect estimation by comparing observed and predicted outcomes: conditions for valid inference and practical illustration

Technical Summary: Treatment Effect Estimation by Comparing Observed and Predicted Outcomes

Problem Statement
New medical treatments often require evaluation for effectiveness and safety before Randomized Controlled Trial (RCT) evidence is available or feasible. While non-randomized methods like regression adjustment or propensity scores exist, a specific approach has emerged in fields such as radiotherapy: estimating the Average Treatment Effect among the Treated (ATT) by comparing observed outcomes under a new treatment against predicted counterfactual outcomes under a standard treatment. These predictions are generated using a model developed on historical or external data where only the standard treatment was administered. While intuitively applied and recognized as a form of standardization, virtual controls, or a g-method, the theoretical conditions required for this approach to yield unbiased causal estimates have not been formally described. This paper addresses the gap between practical application and formal causal inference theory.

Methodology
The authors formalize this estimation approach within the potential outcomes framework for causal inference. The core estimand is the ATT: the average difference between the observed outcome under the new treatment ($T=1$) and the potential outcome under the standard treatment ($T=0$) for the population currently receiving the new treatment.

The estimation procedure involves:

1. Model Development: Fitting a prediction model ($m_{pre}$) on a pre-introduction cohort where all patients received the standard treatment ($T=0$). This model predicts the risk of the outcome based on patient characteristics ($X$) and treatment plan variables specific to the standard treatment ($Z^{(0)}$).
2. Counterfactual Prediction: Applying the fitted model to a post-introduction cohort of patients treated with the new therapy ($T=1$). For these patients, the model predicts what their outcome would have been had they received the standard treatment, utilizing their specific patient characteristics and the hypothetical standard treatment plan variables (which are often available during the selection process for the new treatment).
3. Effect Calculation: The ATT is estimated as the average difference between the observed outcomes in the new treatment group and the predicted counterfactual outcomes.

To illustrate the methodology and its conditions, the authors utilize a case study involving head and neck cancer patients in the Netherlands. The study estimates the benefit of proton therapy (new treatment) versus photon therapy (standard treatment) in reducing dysphagia at 6 months. The model was trained on 750 patients treated with photon therapy (2007–2017) and applied to 93 patients selected for proton therapy (2018–2019) based on a model-based selection algorithm.

Key Conditions for Valid Inference
The paper identifies five sufficient conditions required for the ATT estimate to be unbiased. While largely untestable directly, the authors discuss their implications and provide examples of violations:

1. Transportability: The relationship between covariates and the outcome under the standard treatment must be identical in the pre-introduction (model development) and post-introduction (target) populations. Violations occur if unmeasured confounders (e.g., smoking status, concurrent protocol changes) differ between populations or if dose optimization strategies evolve over time.
2. Ignorability of Treatment Assignment: Within the post-introduction population, the potential outcome under the standard treatment must be independent of the actual treatment received, conditional on the covariates included in the model. This is violated if treatment selection depends on unmeasured factors that also influence the outcome (e.g., specific tumor characteristics or patient quality of life not captured in the model).
3. Consistency: The observed outcome for a patient receiving a specific treatment must match their potential outcome under that treatment. This assumes the treatment is well-defined and that variations in delivery (e.g., clinician skill, equipment differences) do not alter the outcome in ways not captured by the model's covariates.
4. Positivity: The support of covariates in the target population (those receiving the new treatment) must be contained within the support of the covariates in the model development population. Violations occur if the model must extrapolate to patient profiles or treatment plan combinations never seen during development.
5. Correct Model Specification: The functional form of the prediction model must correctly capture the conditional probability of the outcome given the covariates. Misspecification (e.g., ignoring non-linearities or interactions) leads to biased predictions and, consequently, biased ATT estimates.

Results and Validation Strategies
Using the synthetic data from the case study, the authors demonstrate the calculation of the ATT. The estimated effect showed a reduction in dysphagia risk of 0.22 (95% CI: -0.31, -0.13) for proton therapy compared to the predicted photon therapy outcomes.

Crucially, the paper outlines methods to gather supportive evidence for the validity of these conditions, acknowledging that they cannot be directly tested:

• Calibration-in-the-Large (CITL): If a subset of the post-introduction population received the standard treatment, the model's predictions can be compared to their observed outcomes. A difference close to zero supports the plausibility of the conditions.
• Negative Control Populations: Applying the estimation procedure to groups where the treatment effect is expected to be zero.
• Cross-Validation in Treated Groups: In the radiotherapy context, if the relationship between dose and outcome is assumed to hold across treatment types, model performance can be indirectly assessed in the proton-treated group by comparing predicted photon outcomes to observed proton outcomes (with caveats regarding the assumption of dose-outcome equivalence).

Significance and Contributions
The authors claim three primary contributions:

1. Formalization: This is the first work to formally describe this specific "model-based clinical evaluation" strategy within the potential outcomes framework, clarifying the set of sufficient conditions required for valid causal inference.
2. Distinction from Existing Methods: While similar to historical controls, model-based standardization, and g-methods, this approach is distinct because it explicitly relies on a model estimated in a pre-introduction population (introducing a transportability component) and incorporates individualized treatment planning information available prior to treatment.
3. Practical Guidance: By combining theoretical conditions with a realistic case study and providing R code for implementation and sensitivity analysis, the paper offers practical guidance for researchers and clinicians.

The paper concludes that while this approach offers practical advantages—such as not requiring access to individual-level pre-introduction data and flexibility in effect measures—it is susceptible to bias if the identified conditions are violated. Therefore, the authors recommend a systematic evaluation of these conditions, informed by domain expertise and empirical evidence, to ensure the credibility of treatment effect estimates in non-randomized settings.