Comparing Missing Data Methods for Estimating Average Treatment Effects Under Time-Varying Confounding: A Simulation Study

Technical Summary: Comparing Missing Data Methods for Estimating Average Treatment Effects Under Time-Varying Confounding

Problem Statement
In real-world causal inference, data is frequently incomplete due to loss to follow-up, dropout, or non-response. Standard statistical approaches often rely on complete-case analysis, which can yield biased and inefficient estimates when data is not Missing Completely at Random (MCAR). While methods exist to handle missing data and to adjust for confounding (e.g., propensity score weighting), few studies have examined how imputation methods perform specifically under time-varying confounding with binary variables. Furthermore, there is limited understanding of how the missingness mechanism (MCAR, MAR, MNAR), missing rate, location of missingness (treatment vs. outcome), and sample size jointly affect the validity of Average Treatment Effect (ATE) estimates and the underlying identifiability conditions (exchangeability, positivity, and consistency).

Methodology
The authors conducted a simulation study using a full-factorial design to compare four missing data handling strategies:

1. Complete-case analysis: Restricting analysis to subjects with fully observed data.
2. Single mode imputation: Replacing missing values with the most frequent observed value.
3. Stratified hot deck imputation: Imputing missing values using a random donor from the same stratum defined by observed covariates.
4. Multiple Imputation by Chained Equations (MICE): Generating multiple datasets via iterative regression and combining estimates using Rubin's rules.

Simulation Design:

• Data Generation: Synthetic longitudinal datasets were generated based on a vaccine-efficacy setting with time-varying treatment ($A_0, A_1, A_2$) and survival outcomes ($Y_0, Y_1, Y_2, Y_3$). The process included four confounders: comorbidities, age, area-level deprivation, and sex.
• Parameters Varied:
  • Confounding strength ($\rho$): -0.1, -0.4, -0.7.
  • Sample size ($n$): 1,000 and 5,000.
  • Missing rate: 20% and 50%.
  • Missingness mechanism: MCAR, MAR (in treatment or outcome), and MNAR.
  • Missingness location: Treatment variables ($A_1, A_2$) vs. Outcome variables ($Y_1, Y_2$).
• Analysis: ATEs were estimated using logistic regression with Inverse-Probability Treatment Weighting (IPTW). Performance was evaluated across 48 scenarios (500 replications each) using absolute bias, 95% confidence interval coverage, and empirical standard error. The prevalence of extreme propensity scores was used as a proxy for positivity violations.

Key Results

• Method Performance: Multiple Imputation (MICE) consistently outperformed other methods, achieving the lowest bias, highest coverage, and lowest empirical standard error across most scenarios. It remained relatively robust even under MNAR, though all methods showed some deterioration.
  • Complete-case analysis showed low bias under MCAR but suffered from poor and highly variable coverage under non-random missingness. It was particularly sensitive to strong confounding.
  • Single mode imputation performed poorly, especially under MAR, likely due to the distortion of binary variable distributions.
  • Stratified hot deck imputation performed better than complete-case and single mode imputation but yielded inferior coverage compared to MICE.
• Impact of Parameters:
  • Missingness Mechanism: This was the primary driver of bias and empirical standard error. Bias increased progressively from MCAR to MAR to MNAR.
  • Confounding Strength: Strong confounding ($\rho = -0.7$) significantly reduced confidence interval coverage, particularly for complete-case and single mode imputation, suggesting that missing data methods that fail to preserve variable relationships hinder the ability of propensity scores to restore exchangeability.
  • Sample Size: Smaller samples ($n=1,000$) were associated with higher bias and empirical SE, likely due to finite-sample instability in variance estimation rather than estimator bias.
  • Missingness Location: Coverage for MICE and hot deck imputation decreased when missingness occurred in treatment variables rather than outcome variables, attributed to the sparse distribution of the outcome variables limiting imputation information.
• Positivity Violations: Extreme propensity scores (indicating positivity violations) were concentrated in MNAR scenarios, high missing rates (50%), and small sample sizes.

Significance and Conclusions
The study concludes that the missingness mechanism and the choice of missing data method are the dominant determinants of ATE estimation performance. The authors advocate for a general preference for Multiple Imputation (MICE) over simpler ad hoc methods (complete-case, single mode, hot deck) when handling missing treatment or outcome data in propensity-score-based causal analyses, as MICE is the only method that maintained acceptable coverage under MNAR.

However, the authors note that method performance is highly context-dependent. Even sophisticated methods like MICE degrade when missingness affects variables with highly skewed distributions (such as the survival outcomes in this study). The study highlights that strong confounding can compromise coverage even when confounders are fully measured, particularly if the missing data method fails to preserve the necessary relationships for adjustment.

Limitations
The authors acknowledge several limitations: the study introduced missingness via a single mechanism at a time rather than complex, mixed patterns (e.g., intermittent dropout); positivity violations were approximated via extreme propensity scores; and the parameter space was constrained by computational costs. Additionally, each method was implemented with a fixed configuration (e.g., MICE with $m=5$), which may not reflect all practical implementations.

Future Directions
The paper suggests future research should explore doubly robust and machine-learning-based imputation methods under time-varying confounding, apply these comparisons to real-world datasets with complex missingness patterns, and systematically investigate the interaction between positivity violations and missingness mechanisms.