Planning for gold: Hypothesis screening with split samples for valid powerful testing in matched observational studies

1. Problem Statement

Observational studies are essential for causal inference when randomized controlled trials are infeasible. However, they suffer from unmeasured confounding, where hidden biases can invalidate causal conclusions.

• The Challenge: Researchers often face a "multiple testing" problem with many potential outcomes ($L$ outcomes). Standard corrections (like Bonferroni) control the Family-Wise Error Rate (FWER) but suffer from low statistical power, especially when the sample size is split or when controlling for high levels of potential bias ($\Gamma$).
• The Trade-off: While splitting data into a "planning sample" (to design the study) and an "analysis sample" (to test hypotheses) ensures honest inference, it traditionally reduces the sample size available for testing, leading to a loss of power. Previous methods for selecting hypotheses in the planning sample (e.g., choosing the smallest $p$-value) were often heuristic, limited to single outcomes, or failed to account for the variability of the selection process in finite samples.
• Goal: Develop a method to select a subset of outcomes in the planning sample that are most likely to be significant in the analysis sample, specifically prioritizing outcomes robust to hidden bias, without sacrificing the validity of the final inference.

2. Methodology

The authors propose a Split-Sample Hypothesis Screening framework, specifically introducing the Sens-Val Method (Method 2).

Core Concepts

• Data Splitting: The dataset is randomly partitioned into a planning sample ($G_{plan}$) and an analysis sample ($G_{analysis}$).
• Sensitivity Value ($\Gamma^*$): Instead of using asymptotic "design sensitivity," the method uses the sensitivity value, defined as the smallest level of unmeasured confounding ($\Gamma$) required to render a specific outcome's $p$-value insignificant. This is a sample-based random variable that captures finite-sample behavior.
• Transformed Sensitivity Value ($\kappa$): The authors define $\kappa = \Gamma^*/(1+\Gamma^*)$ to map the sensitivity value to the unit interval, facilitating normal approximation.

The Sens-Val Procedure (Method 2)

1. Planning Phase:
   • Compute the sensitivity value ($\kappa_{plan}$) for each of the $L$ outcomes in the planning sample.
   • Use bootstrapping on the planning sample to estimate the variability ($\hat{\sigma}_F$) of the sensitivity value. This is crucial because the sensitivity value is a random variable with non-negligible variance in finite samples.
2. Predictive Interval Construction:
   • Using a derived Central Limit Theorem (Theorem 1) and Edgeworth expansions, the method constructs a predictive interval for what the sensitivity value of an outcome will be in the analysis sample ($\kappa_{analysis}$).
   • The interval accounts for the difference in sample sizes between the planning and analysis sets and the estimated variability from the bootstrap.
3. Selection Rule:
   • An outcome is selected for testing in the analysis sample if the lower bound of its predictive interval for $\kappa_{analysis}$ exceeds the researcher's pre-specified bias tolerance threshold ($\kappa_{con} = \Gamma_{con}/(1+\Gamma_{con})$).
   • This ensures that only outcomes likely to remain significant even under the specified level of hidden bias are carried forward.
4. Analysis Phase:
   • Test the selected outcomes in the analysis sample.
   • Apply standard FWER control (e.g., Bonferroni) only to the selected subset.

Theoretical Guarantees

• Validity: Proposition 1 proves that the FWER is controlled at level $\alpha$ regardless of the selection rule used in the planning sample, provided the selection is independent of the analysis sample.
• Power: Theorem 3 demonstrates that the method has high local asymptotic power, meaning it effectively selects non-null outcomes even when the treatment effect is small (barely above the detection threshold).
• Superiority: Proposition 2 proves that with appropriate hyperparameter tuning, Sens-Val always selects at least as many (and often more) true signals as a "Naïve" method (which simply selects outcomes with $p < \alpha$ in the planning sample).

3. Key Contributions

1. Novel Screening Mechanism: The paper moves beyond heuristic selection (like picking the smallest $p$-value) to a statistically rigorous framework based on the sensitivity value and its estimated variability.
2. Finite Sample Robustness: By utilizing bootstrapping and Edgeworth expansions, the method accounts for the randomness of the sensitivity value in finite samples, unlike previous approaches relying solely on asymptotic design sensitivity.
3. Handling Unmeasured Confounding: The method explicitly optimizes for robustness against hidden bias. It prioritizes outcomes that are less sensitive to unmeasured confounders, making the final causal claims more credible.
4. Theoretical Advancement: The authors derive new Edgeworth expansions for the sensitivity value, showing that the bootstrap approximation achieves $O(I^{-1})$ accuracy, superior to the standard $O(I^{-1/2})$ normal approximation.
5. Practical Application: The method is applied to a real-world dataset regarding the 1998 Bangladesh floods, demonstrating its utility in complex, multi-dimensional observational studies.

4. Results

The authors conducted extensive simulations and a real-data application:

• Simulation Studies:
  • Power vs. Bias: Sens-Val consistently outperformed both the "Naïve" split-sample method and full-sample Bonferroni correction, particularly at higher levels of $\Gamma_{con}$ (stricter control for bias).
  • Sparse Signals: In settings with many outcomes and few true signals (sparse regime), Sens-Val maintained high power by effectively filtering out null hypotheses, whereas full-sample Bonferroni suffered severe power loss due to multiple testing penalties.
  • Oracle Comparison: The performance of Sens-Val approached that of an "Oracle" (which knows the true non-null outcomes beforehand), especially in moderate sample sizes.
• Bangladesh Flood Application:
  • The study analyzed 93 outcomes related to food security, health, and economics.
  • Findings: Sens-Val identified significant effects (e.g., increased illness duration, reduced water access via tubewells, increased food credit usage) that the Naïve method and full-sample Bonferroni missed at higher bias tolerance levels ($\Gamma_{con} \ge 2.0$).
  • Robustness: Even at very high bias levels ($\Gamma_{con} = 38$), Sens-Val still identified key outcomes (e.g., flood depth, repair costs), whereas other methods failed to reject any hypotheses.

5. Significance

• Methodological Shift: This paper provides a solution to the "open problem" of how to select hypotheses in observational studies when unmeasured confounding is a concern. It validates the use of data splitting not just for honesty, but as a tool to increase power by filtering out weak or bias-sensitive signals.
• Causal Inference Rigor: By prioritizing outcomes with high sensitivity values, the method ensures that the reported causal effects are those most resilient to hidden biases, strengthening the credibility of observational findings.
• Practical Utility: The method is computationally feasible (linear in the number of outcomes) and provides a clear, data-driven protocol for researchers to design studies that balance exploration (planning) and confirmation (analysis) without compromising statistical validity.

In summary, "Planning for Gold" offers a robust, theoretically grounded framework for "mining" observational data for true causal signals while rigorously guarding against the pitfalls of hidden bias and multiple testing.