Optimal two-phase sampling designs for generalized raking estimators with multiple parameters of interest

1. Problem Statement

Large observational datasets, particularly Electronic Health Records (EHRs), are invaluable for medical research but suffer from missing data, measurement error, and misclassification. Two-phase sampling is a standard solution: inexpensive, error-prone data are collected for a large Phase 1 cohort, while high-quality, expensive validation data are collected for a smaller Phase 2 subsample.

While optimal designs for Inverse Probability Weighted (IPW) estimators and Generalized Raking (GR) estimators exist for single-parameter settings, there is a gap in methodology for multiple parameters of interest (e.g., studying multiple disease outcomes or predictors simultaneously).

• The Challenge: Existing methods often optimize for one parameter at a time or use ad-hoc strategies (e.g., splitting the sample equally).
• The Specific Gap: It is unclear how optimal designs for GR estimators differ from IPW estimators when multiple parameters are involved, and how to allocate a fixed Phase 2 budget across multiple parameters to minimize the total variance of the estimates.

2. Methodology

The authors propose a framework for adaptive, multiwave sampling where Phase 2 data collection occurs in sequential waves. As data accumulates, the sampling design for subsequent waves is updated based on newly estimated parameters.

A. Estimation Framework

• IPW Estimators: Standard design-based estimators using sampling weights.
• Generalized Raking (GR) Estimators: These calibrate IPW weights using auxiliary variables (Phase 1 data) to match known population totals. GR is asymptotically equivalent to the optimal Augmented IPW (AIPW) estimator and is more efficient when Phase 1 variables are correlated with the target variables.
• Influence Functions: The authors rely on the asymptotic linearity of estimators, representing them as sums of influence functions ($h_i$). The goal is to minimize the variance of these sums.

B. Proposed Allocation Strategies

The paper compares four main strategies for allocating the Phase 2 sample size ($n$) across $K$ strata and $P$ parameters:

1. Independent Allocation (Simultaneous): The total sample $n$ is divided among $P$ parameters. For each parameter, a Neyman or Neyman-Wright allocation is calculated independently, and the stratum sample sizes are summed.
2. Sequential Allocation: The sample is divided into waves. In each wave, the allocation is optimized for one specific parameter (cycling through parameters).
3. Prioritized A-Optimality (Continuous): Minimizes the weighted sum of variances (Trace of the covariance matrix) across all parameters. The continuous solution follows a Neyman-like allocation where the standard deviation for stratum $k$ is replaced by $\sqrt{\sum a_p \sigma^2_{p,k}}$, where $a_p$ are priority weights.
4. Integer-Valued A-Optimal Allocation (Novel Contribution): The authors derive an exact, integer-valued algorithm extending Wright's (2017) method. This ensures sample sizes are whole numbers and respects constraints ($2 \le n_k < N_k$), which is crucial when strata are small.

C. Extension to Generalized Raking

A critical methodological insight is that the optimal design for GR differs from IPW when multiple parameters are present.

• For GR, the optimal allocation depends on the residuals of the influence functions regressed on the raking variables.
• If the correlation between Phase 1 raking variables and the true influence functions varies significantly across parameters, the optimal design for GR will diverge from the IPW design. The authors propose using GR-optimal allocation (based on residuals) rather than IPW-optimal allocation when using GR estimators.

3. Key Contributions

1. Multi-Parameter A-Optimality: Derivation of a novel, exact integer-valued A-optimal allocation algorithm for multiple parameters, extending Wright's single-parameter method.
2. GR vs. IPW Design Divergence: Demonstration that in multi-parameter settings, the optimal sampling design for GR estimators can differ substantially from that of IPW estimators. Optimizing for IPW does not guarantee efficiency for GR when parameters have different relationships with auxiliary variables.
3. Adaptive Multiwave Framework: Formalization of simultaneous and sequential adaptive strategies for multiwave sampling, showing how to update designs as validation data accumulates.
4. Practical Implementation: Integration of these methods into the R package optimall, making the A-optimal allocation for raking estimators accessible to practitioners.

4. Results

The authors evaluated these strategies through extensive simulations (10 scenarios varying correlation and measurement error) and a real-world data example.

Simulation Findings

• GR Superiority: GR estimators consistently outperformed IPW estimators across all designs. The efficiency gain was most pronounced under non-optimal (case-control) designs.
• A-Optimality Performance: The A-optimal design (Strategy 5) achieved the best or near-best empirical relative efficiency (ERE) in almost all scenarios.
  • It minimized the sum of variances better than independent or sequential allocation.
  • In scenarios with high correlation and low measurement error, A-optimality provided significant gains over case-control sampling (up to 22% efficiency gain).
• Sequential vs. Simultaneous: Sequential strategies often optimized the last parameter at the expense of earlier ones. Simultaneous independent allocation performed well but was generally slightly less efficient than A-optimality.
• Measurement Error: As measurement error increased, the efficiency gains from optimal allocation and raking diminished, as Phase 1 variables became less informative.

Real-World Data Example (Vanderbilt HIV Study)

• Context: Estimating the effect of CD4 count on two outcomes: Death and AIDS-defining events (ADE).
• Measurement Error Disparity: The Phase 1 EHR data had high error for ADE (34.5% time error) but low error for Death.
• Key Finding:
  • GR reduced variance for the "Death" parameter by 88–95% (due to high correlation between Phase 1 and Phase 2 data) but only 12–22% for "ADE."
  • Design Trade-off: An IPW-optimal design (which ignores the raking variable's specific impact) failed to balance the efficiency gains. The GR-based A-optimal design correctly identified that raking would handle the "Death" parameter efficiently, so it allocated more sampling resources to the "ADE" parameter (which needed more sampling to overcome poor Phase 1 proxies).
  • This resulted in a significantly lower total variance compared to other strategies.

5. Significance

• Practical Guidance: The paper provides actionable strategies for researchers designing two-phase studies with multiple outcomes. It argues against ad-hoc sample splitting in favor of principled A-optimality.
• Efficiency Gains: By using A-optimality tailored to GR estimators, researchers can achieve substantial reductions in variance (and thus cost or sample size requirements) compared to traditional case-control or single-parameter optimized designs.
• Handling Data Quality Issues: The work highlights that when data quality varies across parameters (e.g., some variables are well-measured in EHRs while others are not), a unified design strategy is essential. The proposed A-optimal design automatically balances these trade-offs.
• Software Availability: The implementation in the optimall R package ensures these advanced designs can be readily applied in future biomedical research.

In conclusion, the paper establishes that independent optimization for each parameter is suboptimal compared to simultaneous A-optimality, and that Generalized Raking requires its own specific design optimization in multi-parameter settings to fully realize efficiency gains.