Exploration, Confirmation, and Replication in the Same Observational Study: A Two Team Cross-Screening Approach to Studying the Effect of Unwanted Pregnancy on Mothers' Later Life Outcomes

1. Problem Statement

The paper addresses a fundamental challenge in observational research: the tension between Exploratory Data Analysis (EDA), Confirmatory Data Analysis (CDA), and Replication.

• The Dilemma: EDA is essential for generating hypotheses and identifying outliers, but it risks data dredging and inflated Type I errors (false positives). CDA and replication are required for scientific rigor but typically require separate datasets or pre-specified protocols that preclude EDA.
• The Constraint: In many unique, high-value observational studies (like the Wisconsin Longitudinal Study), only a single dataset is available. Researchers cannot split the study into separate discovery and replication cohorts without sacrificing statistical power or the ability to perform EDA on the full dataset.
• The Specific Context: The authors investigate the long-term effects of unwanted pregnancies carried to term on mothers' later-life outcomes (mental health, physical health, economic well-being, life satisfaction). Previous studies were limited to short-term effects or single outcomes (e.g., depression) using linear regression, lacking a comprehensive, rigorous causal inference framework that accounts for hidden bias and multiple testing.

2. Methodology: Two Team Cross-Screening

The authors propose a novel framework called Two Team Cross-Screening to perform EDA, CDA, and replication within a single dataset while controlling the Family-Wise Error Rate (FWER).

Core Procedure

1. Pre-Analysis Discussion: All researchers meet to define the variable for splitting the data, the set of potential outcomes, and confounding control strategies. Crucially, they agree on the splitting factor to ensure that the two resulting subgroups have different sources of potential hidden bias.
2. Data and Team Splitting:
   • The researchers divide into two independent teams (Team A and Team B).
   • The data is split into two disjoint sets ($I_1$ and $I_2$) based on the pre-agreed factor.
   • Blinding: Team A analyzes only $I_1$; Team B analyzes only $I_2$. Neither team sees the other's data during the exploration phase.
3. Exploratory Phase (EDA):
   • Team A performs EDA on $I_1$ to design a confirmatory testing plan for $I_2$.
   • Team B performs EDA on $I_2$ to design a confirmatory testing plan for $I_1$.
   • Teams can generate unanticipated hypotheses and select optimal test statistics (e.g., specific U-statistics) based on their exploration.
   • Error Budgeting: The total FWER target $\alpha$ (e.g., 0.05) is split equally ($\alpha/2$) between the two teams.
4. Confirmatory Phase (CDA):
   • Team A executes its plan on Team B's data ($I_2$).
   • Team B executes its plan on Team A's data ($I_1$).
5. Reporting and Inference:
   • Global Null Rejection: If either team rejects a null hypothesis in the confirmatory phase, the global null (no effect across the whole dataset) is rejected. The overall FWER is controlled at $\alpha$ via the Bonferroni inequality.
   • Replicable Findings: If both teams reject the same null hypothesis, the finding is deemed replicable. The probability of making at least one false replicability claim is controlled at $\alpha$ (using the method of Bogomolov and Heller, 2018).

Application to the Study

• Dataset: Wisconsin Longitudinal Study (WLS), tracking high school graduates from 1957.
• Treatment: Unwanted pregnancy (identified at age 36 based on retrospective questions).
• Splitting Factor: Religious Affiliation (Catholic vs. Non-Catholic).
  • Rationale: The authors hypothesized that the reasons for an unwanted pregnancy differ between groups. Catholics might carry unwanted pregnancies due to religious opposition to contraception, while non-Catholics might do so due to contraceptive failure or lack of use. This ensures that if a causal effect is found in both groups, it is robust against specific unmeasured confounders unique to one group.
• Matching: Used Risk-Set Matching to handle time-varying confounders (e.g., education levels changing after the pregnancy). This matches treated women to controls based on covariates observed up to the time of the pregnancy.
• Analysis Plans:
  • Team A (Catholics): Explored Catholic data, found significant effects on depression, self-acceptance, divorce, and additional unwanted pregnancies. Designed a sequential gatekeeping plan for Non-Catholics.
  • Team B (Non-Catholics): Explored Non-Catholic data, found strong evidence for "low-positive affect" and effect modification by age. Designed a plan for Catholics.

3. Key Results

• Global Null Findings: The study successfully rejected the global null for several outcomes, confirming that unwanted pregnancies have long-term negative effects on mothers.
  • Depression: Confirmed higher depression scores.
  • Self-Acceptance: Confirmed lower self-acceptance.
  • Life Events: Confirmed higher rates of divorce/separation and subsequent unwanted pregnancies.
• Replicability:
  • No Replicable Findings: While Team A's plan found significant effects in the Non-Catholic data, Team B's plan failed to reject the null in the Catholic data (specifically, the "low-positive affect" hypothesis was not significant for Catholics).
  • Interpretation: The lack of replicability suggests potential effect modification by religious affiliation or differences in how "unwanted pregnancy" manifests in these groups. The authors note that in the historical context of the data (pre-Roe v. Wade), the structural exposure was similar, but the psychological or social drivers may differ.
• Comparison to Other Methods:
  • Compared to Automated Cross-Screening (which requires pre-specified rules without EDA) and Holm's method on full data, the Two Team approach allowed the researchers to:
    1. Identify and discard uninterpretable hypotheses (e.g., an income increase driven solely by survivor benefits) that automated methods would have tested.
    2. Discover new, unanticipated hypotheses (e.g., specific subscales of depression, divorce rates) that were not in the original pre-specified list.

4. Key Contributions

1. Methodological Innovation: The paper introduces a rigorous framework to integrate EDA, CDA, and Replication into a single observational study, solving the "single dataset" constraint.
2. Error Control: It provides theoretical guarantees for controlling both the global FWER and the False Coverage Rate (FCR) for replicability claims, even when hypotheses are generated adaptively.
3. Efficiency: By allowing teams to explore their specific data partitions, the method optimizes the allocation of the error budget ($\alpha$) toward the most promising hypotheses, avoiding the waste of testing uninformative variables.
4. Application to Causal Inference: The study provides the most comprehensive evidence to date on the long-term consequences of unwanted pregnancies, utilizing advanced causal inference techniques (Risk-Set Matching, Sensitivity Analysis for hidden bias $\Gamma$).

5. Significance and Implications

• Scientific Rigor: The approach sets a new standard for observational studies where data is scarce or unique. It prevents the "p-hacking" associated with EDA while retaining the creativity and hypothesis-generation benefits of exploration.
• Policy Relevance: The findings highlight that unwanted pregnancies carried to term have lasting negative impacts on mental health (depression, self-acceptance) and family stability (divorce, subsequent unwanted pregnancies). This underscores the need for policies supporting women in situations where abortion is restricted or unavailable.
• Future Directions: The authors suggest this framework could be extended to randomized trials (to allow EDA within strict FDA protocols) and genome-wide association studies (GWAS) to optimize discovery and replication cohorts. They also discuss potential extensions to multi-team designs (3+ teams) and random splitting strategies to handle effect modification.

In summary, the paper demonstrates that by splitting researchers and data into independent, cross-screening teams, it is possible to achieve the "holy grail" of observational research: exploring data to find the truth, confirming it rigorously, and replicating it—all within a single dataset.