Transporting treatment effects by calibrating large-scale observational outcomes

Technical Summary: Transporting Treatment Effects by Calibrating Large-Scale Observational Outcomes

Problem Statement
The paper addresses the challenge of transporting treatment effect estimates from a small, high-quality experimental dataset (Randomized Controlled Trial, or RCT) to a large, target observational dataset. This setting is common when experimentation is expensive or infeasible on the specific target population, necessitating the use of auxiliary observational data.

Standard causal transportation methods rely on two identifying assumptions: mean exchangeability (conditional means of potential outcomes are the same across populations given covariates) and positivity (the support of the covariate distribution in the target population is contained within the experimental population). The paper highlights that positivity violations are likely when the experimental dataset is small relative to the covariate dimension (the "curse of dimensionality"). When positivity fails, the standard transported Average Treatment Effect (ATE) is not well-defined for covariate regions present in the observational data but absent in the experimental data. Furthermore, existing methods that attempt to handle positivity violations (e.g., trimming or downweighting) often alter the estimand in ways that may not align with scientific goals or introduce instability.

Methodology
The authors propose a simple, two-stage estimation and inference procedure that avoids inverse propensity weighting and remains valid even when positivity does not hold.

1. Calibration via Ordinary Least Squares (OLS):

   • The method assumes the observational dataset contains a "treatment-control contrast" ($\Delta(X)$), defined as the difference in conditional mean outcomes between treated and control units in the observational data ($\Delta(x) = E_{obs}[Y|X=x, Z=1] - E_{obs}[Y|X=x, Z=0]$).
   • The authors posit that the true Conditional Average Treatment Effect (CATE), $\mu(X)$, from the experimental data can be approximated by a linear transformation of this observational contrast. Specifically, they define a function class $\mathcal{F}$ of linear transformations of a basis expansion $\psi(\Delta(X))$.
   • They estimate the coefficients $\bar{\beta}$ by regressing the experimental outcomes $D_i$ (the treatment effect) onto $\psi(\hat{\Delta}(X_i))$ using OLS. Here, $\hat{\Delta}$ is estimated from the observational data, potentially using flexible machine learning methods (e.g., causal forests).
   • The resulting estimator $\bar{\mu}(x) = \hat{\bar{\beta}}^\top \psi(\hat{\Delta}(x))$ serves as a calibrated estimate of the CATE.

2. Aggregation:

   • The final estimator for the transported effect, $\hat{\bar{\tau}}$, is the sample average of these calibrated CATE estimates over the covariates in the large observational dataset.
   • To ensure valid inference and avoid overfitting, the authors employ cross-fitting: the observational data is split into folds; $\hat{\Delta}$ is estimated on out-of-fold data, and the calibration regression is performed on the experimental data.

3. Inference:

   • The paper derives an asymptotic variance estimator that accounts for the calibration error (from the OLS step). Under the regime where the observational sample size $N$ grows sufficiently faster than the experimental sample size $n$ ($N \gg n$), the variance is dominated by the calibration error, allowing for asymptotically valid confidence intervals without estimating propensity scores.

Key Contributions

• Alternative Estimand ($\bar{\tau}$): The paper introduces a new estimand, $\bar{\tau} = E_{obs}[\bar{\mu}(X)]$, which represents the average of the best mean-square approximation of the true CATE within the specified function class $\mathcal{F}$, weighted by the observational covariate distribution.
  • Crucially, $\bar{\tau}$ remains identified even without positivity. If the calibration model is well-specified (i.e., $\mu \in \mathcal{F}$), $\bar{\tau}$ equals the standard transported ATE. If misspecified, it converges to a projection estimand, which can be interpreted as a weighted average treatment effect.
• Robustness to Positivity Violations: Unlike existing methods (e.g., AIPSW, collaborative estimators) that rely on propensity scores and fail or become unstable when positivity is violated, the proposed method does not require estimating inverse propensity weights. It remains stable even when the experimental and observational covariate supports do not overlap.
• Semiparametric Efficiency: The authors establish that under a "nested trial" framework where the ratio of experimental to observational sample sizes vanishes ($n/N \to 0$), their estimator achieves the semiparametric efficiency bound for the projected estimand $\bar{\tau}$. This bound is finite even when the efficiency bound for the standard transported ATE is infinite due to positivity violations.
• Simplicity and Practicality: The method relies on standard OLS and flexible CATE learners, avoiding the complex weighting schemes and stability issues associated with high-dimensional propensity score estimation.

Results

The paper validates the methodology through numerical simulations and a real-world data example:

1. Univariate Simulation: In settings with covariate shift (but no strict positivity violation), the proposed estimator ($\hat{\bar{\tau}}$) showed lower Mean Squared Error (MSE) and more stable confidence interval coverage (approx. 92-93%) compared to AIPSW and collaborative estimators as the degree of covariate shift increased. The competing estimators exhibited significant bias and coverage degradation.
2. Multivariate Simulation: In high-dimensional settings ($d=10$), the proposed method maintained the lowest MSE and reliable coverage. In contrast, the AIPSW estimator showed high variance and poor coverage, particularly when the observational covariate distribution had higher variance (exacerbating positivity violations).
3. Crop Rotation Case Study: Using data from Kluger et al. (2022) on maize yields, the authors combined 611 field experiments with 1.31 million satellite-based yield estimates.
   • The experimental data had sparse spatial coverage (missing entire states like Indiana and Michigan), leading to severe positivity violations.
   • The AIPSW estimator was extremely unstable, with estimates spanning several orders of magnitude across simulations.
   • The collaborative estimator was more stable but still exhibited higher variance than the proposed method.
   • The proposed estimator ($\hat{\bar{\tau}}$) demonstrated the lowest variance and highest stability, with a 96% empirical coverage rate for its confidence intervals.

Significance and Claims

The paper claims that its methodology provides a theoretically principled and empirically robust solution for transporting treatment effects in the common regime of "small experiment, large observation."

• Theoretical Validity: The authors demonstrate that their estimator is consistent for a well-defined estimand ($\bar{\tau}$) even when the standard transported ATE is unidentified. They prove that this estimator achieves semiparametric efficiency in the limit of large observational data relative to experimental data.
• Practical Utility: The method offers a "naive" but effective alternative to complex weighting schemes. By avoiding inverse propensity weighting, it circumvents the instability caused by positivity violations, making it particularly suitable for real-world applications where experimental data is limited and spatially or demographically sparse.
• Interpretability: Even when the calibration model is misspecified, the resulting estimand retains interpretability as a weighted average treatment effect, similar to literature on ATE estimation under positivity violations in single observational studies.

The authors conclude that while their method assumes i.i.d. observations (which may be a simplification for spatial/temporal data like the crop rotation example), it offers a significant improvement in precision and stability over existing proposals for transporting treatment effects under positivity violations.