Powerful Multivariate Sensitivity Analysis via Sample Splitting in an Observational Study of the Effects of Poverty on Cardiovascular Disease Risk Factors

Technical Summary: Powerful Multivariate Sensitivity Analysis via Sample Splitting

Problem Statement
In observational studies assessing the causal effects of an exposure on multiple outcomes (multivariate settings), researchers often seek to test a global null hypothesis (that the exposure has no effect on any outcome). A common strategy to improve power and robustness against unmeasured confounding is to construct a linear combination of outcome-specific test statistics. However, existing methods for finding the optimal linear combination—such as those using Scheffé projections or constrained variants (e.g., Cohen et al., 2020; Rosenbaum, 2016)—require corrections for multiple testing. These corrections rely on reference distributions (e.g., chi-bar-squared) whose critical values scale with the number of outcomes ($K$). As $K$ increases, these critical values become increasingly stringent, eroding statistical power and making it difficult to reject the global null hypothesis even when strong evidence exists. Furthermore, the computational burden of evaluating these distributions grows with $K$.

Methodology
The authors propose a novel sensitivity analysis framework that utilizes sample splitting to decouple the selection of the linear combination from the inference step. The procedure operates as follows:

1. Data Partitioning: The matched dataset is randomly split into a planning (pilot) sample and an analysis sample.
2. Planning Phase: On the planning sample, the researcher solves an optimization problem to identify the linear combination of test statistics (constrained to the probability simplex, $\Delta_+$) that maximizes the sensitivity to unmeasured bias (i.e., minimizes the worst-case p-value under the Rosenbaum sensitivity model). This yields an estimated optimal weight vector, $\hat{\lambda}_{plan}$.
3. Inference Phase: The vector $\hat{\lambda}_{plan}$ is treated as pre-specified for the analysis sample. The researcher conducts the sensitivity analysis on the analysis sample using this fixed linear combination.
4. Statistical Justification: Because the combination is pre-specified relative to the analysis data, the test statistic follows a standard Normal distribution under the null hypothesis, rather than a complex chi-bar-squared distribution dependent on $K$. This allows the use of a critical value (e.g., 1.645 for $\alpha=0.05$) that is independent of the number of outcomes.

Theoretical Contributions
The paper addresses a critical theoretical gap: sample splitting typically introduces a "minimax gap" where the researcher must commit to a strategy before nature (the unmeasured confounder) acts, potentially leading to strictly lower power than full-sample methods (as noted by Fogarty and Small, 2016).

• Minimax Theorem: The authors prove a novel minimax theorem (Theorem 1) showing that for the specific case where the set of linear combinations is the probability simplex ($\Delta_+$), the order of optimization (researcher first vs. nature first) does not matter. The minimax equality holds:
  $$\min_{\varrho \in P_\Gamma} \sup_{\lambda \in \Delta_+} F(\lambda, \varrho) = \sup_{\lambda \in \Delta_+} \min_{\varrho \in P_\Gamma} F(\lambda, \varrho)$$
• Asymptotic Power: As a corollary (Theorem 2), they demonstrate that the design sensitivity (the asymptotic limit of the sensitivity value) of their sample-splitting approach is identical to that of full-sample alternatives. Thus, the method does not sacrifice asymptotic power despite using only a subset of the data for inference.

Simulation Results
Extensive simulation studies were conducted with $I=300$ matched pairs and varying numbers of outcomes ($K \in \{5, 15, 25\}$) and correlations ($\rho \in \{0, 0.2\}$).

• Finite Sample Power: The sample-splitting method consistently demonstrated higher power than the full-sample implementation of Cohen et al. (2020), particularly as $K$ increased. This is attributed to the avoidance of $K$-dependent critical values.
• Comparison to Oracle: While an "oracle" (knowing the optimal combination a priori) performed best, the sample-splitting approach significantly outperformed the full-sample competitor and approached the oracle's performance in many regimes.
• Correlation Effects: The power advantage of the proposed method was most pronounced when outcomes were uncorrelated or weakly correlated, though it remained superior even with moderate positive dependence.

Application: Poverty and Cardiovascular Disease (CVD) Risk
The method was applied to a study of the effects of poverty (defined as Family Poverty Income Ratio < 1) on CVD risk factors in children and adolescents using NHANES data (1999–2016).

• Study Design: 4,027 matched pairs were split into a pilot sample (25%) and an analysis sample (75%).
• Outcomes: The study examined 9 outcomes including body composition, physical activity, lipids, and tobacco exposure (cotinine levels).
• Findings:
  • The sample-splitting approach rejected the global null hypothesis for all age/gender strata at $\Gamma=1$, whereas the full-sample comparator only rejected the global null for boys aged 8–11.
  • The method identified significant adverse effects of poverty on tobacco exposure (cotinine levels) across all strata and vigorous leisure-time physical activity (LTPA) for adolescents.
  • Robustness: The findings regarding tobacco exposure showed robustness to unmeasured confounding up to $\Gamma \approx 2.25$ (sensitivity value $\approx 3.05$). In contrast, findings regarding physical activity and body composition were sensitive to smaller amounts of bias (failing to reject at $\Gamma \approx 1.05$–$1.15$).
  • Comparison: A full-sample sensitivity analysis on the same data yielded a much lower sensitivity value ($\approx 1.67$) for the overall hypothesis, illustrating the power loss inherent in existing multivariate methods for this dataset.

Significance
The paper claims that its primary contribution is a practical and theoretically grounded framework for conducting powerful multivariate sensitivity analyses in observational studies with many outcomes. By leveraging sample splitting and a new minimax theorem, the method:

1. Recovers Power: It avoids the severe power loss associated with $K$-dependent critical values in high-dimensional settings.
2. Maintains Validity: It provides valid level-$\alpha$ inference without requiring asymptotic corrections that depend on the dimension of the outcome space.
3. Enables Data-Driven Design: It allows researchers to use the pilot sample to refine outcome selection and test statistic choices (e.g., identifying specific dietary components or age-specific effects) without compromising the validity of the final inference.

The authors conclude that while the robustness of their specific findings on poverty and CVD is largely driven by the effect on tobacco exposure, the methodological framework successfully enables the detection of these effects where traditional multivariate sensitivity analyses would likely fail due to lack of power.