Moving toward best practice when using propensity score weighting in survey observational studies

Technical Summary: Moving toward best practice when using propensity score weighting in survey observational studies

Problem Statement
Propensity score weighting is a standard method for estimating causal treatment effects from observational data by addressing confounding due to measured baseline covariates. However, when observational data are derived from complex surveys, the optimal integration of survey weights for population-level causal inference remains a subject of debate. Existing literature offers conflicting recommendations: some suggest using survey weights as covariates in the propensity score model, while others advocate for survey-weighted regression. Furthermore, prior work has largely focused on the Average Treatment Effect (ATE) and the Average Treatment Effect on the Treated (ATT), often neglecting other target populations such as the overlap population. There is a lack of consensus on how to incorporate survey weights across different stages of estimation (propensity score modeling, outcome modeling, and standardization) to ensure unbiased and efficient inference for various estimands under complex sampling designs, particularly when overlap is poor or models are misspecified.

Methodology
The authors propose a unified framework under the balancing weights paradigm to incorporate survey weights into propensity score weighting for survey observational studies.

1. Theoretical Framework:

   • The paper distinguishes between retrospective designs (sampling occurs after treatment assignment, where sampling probabilities may depend on treatment status) and prospective designs (sampling occurs before treatment assignment).
   • It defines a general class of Population Weighted Average Treatment Effects (PWATE), $\tau_h$, characterized by a tilting function $h(X)$ that shifts the covariate distribution from the original population to a target population. This framework encompasses ATE, ATT, Average Treatment Effect on the Control (ATC), and the Average Treatment Effect on the Overlap (PATO).
   • The authors derive weighting estimators that combine balancing weights (to address confounding) with survey weights (to address selection bias). The weights are defined as $\omega_{z,p}(X_i) = h(X_i) / [p_z(X_i) e_{sp}(X_i)]$, where $p_z(X)$ represents treatment-specific sampling probabilities and $e_{sp}(X)$ is the population-level propensity score.

2. Estimators Developed:

   • Propensity Score Weighting (PSW): A standard weighting estimator incorporating survey weights in both the propensity score estimation and the outcome standardization.
   • Augmented Weighting (AW) Estimators: To improve efficiency, the authors develop three augmented estimators that leverage outcome regression models:
     • One-step Moment Estimator (MOM): Uses unweighted outcome models but applies survey weights to standardize predictions and residuals.
     • Clever Covariate Estimator (CVR): Incorporates balancing weights as additional covariates ("clever covariates") in the outcome regression model.
     • Weighted Regression Estimator (WET): Fits the outcome regression model using balancing weights as regression weights.
   • Variance Estimation: Closed-form sandwich variance estimators are derived for all estimators using M-estimation theory, accounting for the uncertainty in both propensity score and outcome model estimation, as well as the survey design.

3. Simulation Design:

   • The authors conducted extensive simulations involving 1,000,000 individuals organized in a multi-stage sampling design.
   • Scenarios varied by: overlap quality (good vs. poor), model specification (correct/misspecified propensity score and outcome models), and sampling mechanism (prospective vs. retrospective where sampling depends on treatment).
   • They compared strategies for incorporating survey weights: using them as regression weights (W.PS) versus as an additional covariate (C.PS).

Key Results

• Incorporation of Survey Weights: The simulations consistently demonstrate that using survey weights as regression weights in the propensity score model (W.PS) yields lower bias, higher efficiency, and better coverage than using them as a covariate (C.PS). This advantage is particularly pronounced under poor overlap and retrospective sampling designs where sampling depends on treatment.
• Performance of Augmented Estimators:
  • When models are correctly specified, all augmented estimators (MOM, CVR, WET) outperform PSW alone.
  • CVR achieves high efficiency under good overlap but exhibits instability and poor coverage under poor overlap or when the outcome model is misspecified.
  • MOM offers robustness but generally lower efficiency compared to WET.
  • WET emerges as the most robust and efficient estimator across all scenarios, including poor overlap and model misspecification. It maintains near-nominal coverage and lower bias compared to other methods.
• Target Populations: The Overlap (PATO) estimand demonstrates superior robustness to poor overlap and model misspecification compared to ATE and ATT, often achieving the lowest bias and highest efficiency, especially when combined with WET and W.PS.

Case Studies
The authors applied these methods to two real-world datasets:

1. ECLS-K (Prospective Design): Analyzing the effect of special education on math achievement. Results showed that WET provided stable estimates across different target populations, while PSW yielded wide confidence intervals.
2. MEPS (Retrospective Design): Analyzing racial disparities in healthcare expenditures. Due to oversampling of minority groups, survey weights were highly variable. WET with W.PS successfully mitigated the instability caused by extreme weights, providing precise estimates for the overlap population where covariate balance was achieved.

Significance and Claims
The paper claims to provide a unified solution for incorporating survey weights into propensity score weighting, extending the balancing weights framework to cover a broader class of estimands (including PATO) and offering closed-form variance estimators for efficient inference.

The authors conclude with specific practical recommendations:

1. Use Survey-Weighted Propensity Scores (W.PS): Survey weights should be used as regression weights in the propensity score model to ensure the sampled covariate distribution aligns with the target population, rather than treating them merely as covariates.
2. Prefer Weighted Regression Estimators (WET): Among augmented methods, WET is recommended for its superior stability and robustness, particularly in scenarios with poor overlap or model misspecification.
3. Consider Overlap Weights (PATO): For policy-relevant inference in populations with limited overlap, targeting the overlap population using overlap weights provides more reliable and efficient estimates.

The authors emphasize that these methods are implemented in the PSweight R package, facilitating their application in complex survey observational studies. They acknowledge limitations regarding assumptions of unconfoundedness and known survey weights, suggesting future work on sensitivity analyses and extensions to multiple treatments and machine learning-based approaches.