On the error control of invariant causal prediction

Technical Summary: On the error control of invariant causal prediction

Problem Statement
Invariant Causal Prediction (ICP), introduced by Peters et al. (2016), is a framework for identifying causal predictors of a response variable using heterogeneous data from multiple environments. The original method guarantees that all discovered predictors are true causal predictors with high probability by controlling the Familywise Error Rate (FWER). While this guarantee is strong, it is often overly conservative, particularly when causal predictors are highly correlated with non-causal ones, leading to few or no discoveries. The paper addresses the question of whether ICP can be equipped with less conservative error guarantees—specifically False Discovery Rate (FDR) control and simultaneous true discovery bounds—to extract more causal information from the data without compromising statistical validity.

Methodology
The authors reformulate ICP as a multiple testing problem. Instead of directly estimating the set of causal predictors, they propose testing the null hypotheses $H^*_{0,i}: i \notin S^*$ (that variable $i$ is not a causal predictor) for all $i \in [m]$.

1. Multiple Testing Formulation:

   • Based on the $p$-values $p_S$ generated by ICP for invariance testing of subsets $S$, the authors construct valid $p$-values for single hypotheses: $p^*_i = \max_{S \subseteq [m]\setminus\{i\}} p_S$.
   • They also define $p$-values for intersection hypotheses $H^*_{0,S}: i \notin S^*$ for all $i \in S$ as $p^*_S = \max_{I \subseteq [m]\setminus S} p_I$.
   • A key theoretical finding is that the original ICP discovery set is exactly equivalent to the set of hypotheses rejected by directly comparing $p^*_i$ to the nominal level $\alpha$ without any multiple testing correction. Consequently, the original ICP method already controls FDR, but simply relaxing the threshold on these specific $p$-values does not yield more discoveries.

2. FDR Control via e-Closure:

   • To achieve non-trivial FDR control (i.e., obtaining more discoveries than the original ICP), the authors adopt the e-Closure principle (Xu et al., 2025).
   • They transform the $p$-values $p^*_S$ into e-values using newly designed p-to-e calibrators.
   • The authors propose the Su calibrator and two adapted step-function versions: Step-Su and Step-LinearSu. These calibrators are tailored to the discrete threshold structure inherent in the e-Closure decision rule for FDR, optimizing the allocation of the unit integral budget to improve power.
   • Applying the e-Closure principle to the resulting e-collection yields a family of discovery sets that satisfy simultaneous FDR control. This means the FDR guarantee holds for all sets in the family simultaneously, allowing users to select a set based on data-dependent criteria while retaining the guarantee.

3. Simultaneous True Discovery Bounds:

   • The authors derive simultaneous true discovery bounds using the closed testing procedure (Marcus et al., 1976; Goeman and Solari, 2011).
   • Due to the monotonicity of the constructed $p^*_S$, the computationally expensive closed testing adjustment step is unnecessary; a hypothesis is rejected if and only if it is locally rejected.
   • This leads to a simplified formula for the simultaneous true discovery bound $t_\alpha(R)$, which provides a high-probability lower bound on the number of true causal predictors in any user-specified set $R$.
   • The authors show that this bound is equivalent to a bound derived directly from the original ICP formulation ($\tilde{t}_\alpha(R)$), providing a clearer statistical interpretation (e.g., admissibility) of the original method's properties.

Key Contributions

• Reformulation: Casting ICP as a multiple testing problem for null causal predictors, revealing that the original ICP method is equivalent to unadjusted testing of specific $p$-values derived from intersection hypotheses.
• Non-Trivial FDR Control: Demonstrating that direct $p$-value adjustment cannot improve upon ICP for FDR control, and successfully introducing the e-Closure principle with novel p-to-e calibrators (Step-Su, Step-LinearSu) to achieve FDR control that yields more discoveries than the original FWER-controlled ICP.
• Simultaneous Bounds: Deriving simultaneous true discovery bounds that recover the original ICP results while providing additional information about the number of true discoveries in arbitrary sets. The method simplifies the standard closed testing procedure by eliminating the need for adjustment steps.
• Equivalence: Proving the equivalence between the simultaneous bounds derived via closed testing and those derived from the original ICP formulation, bridging two different statistical perspectives.

Results

• Simulations: The authors conducted simulations on linear structural equation models with varying signal strengths and nominal levels.
  • The e-Closure methods with adapted calibrators (Step-Su, Step-LinearSu) consistently outperformed the general Su calibrator and the original ICP method in terms of the number of true discoveries, particularly in settings with moderate to high signal strength.
  • Empirical FDRs were confirmed to be below the nominal level across all settings.
  • Simultaneous bounds provided additional insights, such as lower bounds on the number of true causal predictors in the full set of variables, which were not available from the single discovery set of the original ICP.
• Real Data Application: Applied to a dataset on educational attainment of teenagers in the US.
  • The original ICP identified two causal predictors: test score and father's college status.
  • The FDR-controlled e-Closure method identified these two plus a third variable ("ethnicity other"), illustrating the ability to extract more candidates for exploratory analysis.
  • Simultaneous bounds revealed that at least five of eight specific variables in a larger set were likely true causal predictors, offering a richer characterization of the causal structure than the original method.

Significance and Claims
The paper claims that the proposed methods offer a practical alternative to the original ICP by relaxing the error control from FWER to FDR and simultaneous bounds.

• Practical Utility: The authors argue that the original FWER control should not be the default choice because the proposed simultaneous true discovery bounds can recover the FWER-controlled set while providing strictly more information.
• Exploratory vs. Confirmatory: The methods allow researchers to choose between a probability-type guarantee (simultaneous bounds) for confirmatory analysis or an expectation-type guarantee (FDR control) for more exploratory analysis where larger discovery sets are desired.
• Methodological Advancement: The work showcases the utility of e-values and the e-Closure principle in scenarios where standard $p$-value approaches are limited (specifically, where unadjusted $p$-values already control FWER). It demonstrates that rigorous error control can be achieved in ICP with less conservatism, thereby extracting more causal information from heterogeneous data.

The authors remain modest regarding future directions, noting that while their methods apply to linear models, they could potentially be extended to non-linear, sequential, or single-environment settings, and that computational challenges for large $m$ remain a hurdle shared by both the original and proposed methods.