Design-based inference for generalized causal effects in randomized experiments

1. Problem Statement

Randomized experiments increasingly utilize complex outcomes (ordinal, multivariate, prioritized composite endpoints) where the classical Average Treatment Effect (ATE), defined by linear contrasts of means, is insufficient. Researchers often rely on Generalized Causal Effect (GCE) estimands, such as the Mann-Whitney parameter (probabilistic index) or causal net benefit, which are defined through pairwise contrast functions ($w$) rather than linear differences.

While regression adjustment (covariate adjustment) is standard for improving efficiency in linear ATE estimation (e.g., Lin, 2013), its theoretical properties for GCEs defined by nonlinear contrast functions remain unclear. Specifically, three critical gaps exist:

1. Consistency & Efficiency: Does covariate adjustment preserve consistency and guarantee efficiency gains for nonlinear GCEs under model misspecification?
2. Variance Estimation: Standard variance estimators (heteroskedasticity-robust, cluster-robust) rely on specific independence assumptions that may fail due to the complex dependence structure inherent in pairwise comparisons (shared units and reverse comparisons).
3. Unified Framework: There is a lack of a unified design-based inference framework that accommodates both individual pair data and per-unit pair averages for these estimands.

2. Methodology

2.1 Framework and Estimands

The authors operate under the finite-population potential outcomes framework with complete randomization.

• GCE Definition: The target estimand is defined as a U-statistic averaging a contrast function $w$ over all distinct ordered pairs of units:
  $$\lambda(a, 1-a) = \frac{1}{N(N-1)} \sum_{i \neq j} w\{Y_i(a), Y_j(1-a)\}$$
  This formulation covers linear contrasts (ATE), probabilistic indices, and causal net benefits ($\tau(a) = \lambda(a, 1-a) - \lambda(1-a, a)$).

2.2 Regression Estimators

The paper proposes two classes of regression estimators to adjust for baseline covariates ($X$):

1. Individual Pair Level: Regressing the pairwise contrast $W_{i,j}$ on treatment indicators and covariates (or interactions).
   • Neyman-type: Unadjusted.
   • Fisher-type (ANCOVA): Adjusted for covariates.
   • Lin-type: Adjusted for covariates and treatment-covariate interactions.
2. Per-Unit Pair Average Level: Aggregating pairwise contrasts to the unit level ($W_{i, \cdot}$) and regressing these averages on treatment indicators and covariates.

2.3 Theoretical Tools

• U-Statistic Theory: The estimators are treated as order-two U-statistics. The authors utilize Hoeffding decomposition to derive Bahadur representations, separating the estimators into a dominant linear term and a degenerate remainder.
• Finite-Population Asymptotics: Leveraging the Central Limit Theorem for simple random sampling (Li and Ding, 2017) to establish asymptotic normality without assuming a super-population model.
• Variance Estimation: The authors analyze the correlation structure induced by pairwise comparisons, noting that $W_{i,j}$ and $W_{j,i}$ are dependent, and units $i$ and $j$ appear in multiple pairs.

3. Key Contributions and Theoretical Results

3.1 Consistency and Model-Assisted Nature

• Result: All proposed regression estimators (Neyman, ANCOVA, Lin-type) are consistent and asymptotically normal for the GCE, even if the working regression models are arbitrarily misspecified.
• Implication: These estimators are model-assisted, not model-based. They rely on the randomization design for validity, not the correctness of the outcome model.

3.2 The Efficiency Limitation (Key Departure from Linear ATE)

• Finding: Unlike the classical linear ATE setting (where Lin, 2013 proved that fully interacted regression guarantees efficiency gains), covariate adjustment for nonlinear GCEs does not admit a universal efficiency guarantee.
• Mechanism: Under nonlinear contrast functions, the regression coefficient used in the adjustment does not necessarily correspond to the "optimal" coefficient that minimizes the Neyman variance. Consequently, adjustment can theoretically increase variance, though simulations show it often decreases variance when covariates are prognostic.
• Significance: This clarifies that efficiency gains from regression are estimand-dependent.

3.3 Variance Estimation: The Complete Two-Way (CTW) Estimator

• Problem: Standard variance estimators fail in this setting:
  • Heteroskedasticity-Robust (HR): Inconsistent because it ignores pairwise dependence.
  • Cluster-Robust (CR): Inconsistent because it ignores half the correlations.
  • Two-Way (TW) Clustering: Consistent for marginal variances of $\hat{\lambda}(a, 1-a)$ but inconsistent for the covariance between $\hat{\lambda}(a, 1-a)$ and $\hat{\lambda}(1-a, a)$. This leads to anti-conservative inference for the causal net benefit $\hat{\tau}(a)$.
• Solution: The authors propose the Complete Two-Way (CTW) Cluster-Robust Variance Estimator.
  • It explicitly accounts for the two-way dependence (units $i$ and $j$) and the reverse comparisons ($W_{i,j}$ vs. $W_{j,i}$).
  • Theorem: The CTW estimator is consistent for the Neyman variances of both individual pair and per-unit average estimators, as well as the covariance required for net benefit inference.

4. Simulation and Empirical Results

4.1 Simulation Studies

• Consistency: All estimators remained unbiased across various data generating processes (univariate, multivariate, ordinal) even with model misspecification.
• Efficiency:
  • When covariates were prognostic, adjusted estimators (ANCOVA/Lin-type) generally yielded smaller standard errors than unadjusted ones.
  • When covariates were unrelated or noisy, adjusted estimators sometimes performed worse or showed no gain, confirming the lack of a universal efficiency guarantee.
• Variance Estimation:
  • HR and CR estimators resulted in severe under-coverage (e.g., ~15% coverage for 95% CIs).
  • TW estimators provided good coverage for marginal effects but failed for net benefits.
  • CTW estimators achieved nominal coverage (~95%) for all estimands, validating the theoretical consistency.

4.2 Data Example (BestAIR Trial)

• Applied to a trial of CPAP treatment for sleep apnea.
• Outcomes included systolic blood pressure (continuous) and sleepiness (ordinal).
• Result: Adjusted estimators generally produced narrower confidence intervals than unadjusted ones, suggesting the covariates were prognostic. The CTW variance estimator confirmed the statistical significance of the treatment effect (causal net benefit > 0).

5. Significance and Conclusion

This paper provides a rigorous design-based inference framework for a broad class of nonlinear causal effects in randomized experiments. Its primary contributions are:

1. Theoretical Clarification: It establishes that while regression adjustment remains valid (consistent) for nonlinear GCEs, the "universal efficiency gain" property of the classical ATE does not hold.
2. Practical Solution for Inference: It identifies the failure of standard robust variance estimators in pairwise comparison settings and introduces the Complete Two-Way (CTW) estimator as the necessary remedy for valid inference on causal net benefits.
3. Unified Approach: It bridges the gap between U-statistic theory and regression adjustment, offering a flexible toolkit for analyzing complex, non-Gaussian, and multivariate outcomes in clinical and social science experiments.

The work is essential for researchers moving beyond mean differences to more interpretable, distribution-free treatment effect summaries, ensuring that their statistical inference remains valid under the randomization design.