Bayesian Hierarchical Invariant Prediction

1. Problem Statement

The paper addresses the challenge of causal discovery (specifically identifying the set of causal parents of a target variable) in the presence of heterogeneous data.

• Context: Real-world data often comes from distinct "environments" (e.g., different hospitals, cities, or time periods) where the distribution of covariates shifts, but the underlying causal mechanisms may remain invariant.
• Limitations of Existing Methods: The standard approach, Invariant Causal Prediction (ICP), identifies causal predictors by finding subsets of variables where the conditional distribution of the target remains invariant across environments. However, ICP has two major drawbacks:
  1. Computational Scalability: It requires exhaustive testing of all $2^d$ subsets of predictors (where $d$ is the number of predictors), making it intractable for high-dimensional data.
  2. Conservatism: To minimize Type I errors, ICP uses an intersection of valid subsets, often leading to low statistical power (missing true causal predictors) or returning an empty set when heterogeneity is insufficient.
  3. Lack of Uncertainty Quantification: Frequentist approaches do not naturally provide probabilistic measures of uncertainty for the identified causal sets.

2. Methodology: Bayesian Hierarchical Invariant Prediction (BHIP)

The authors propose BHIP, which reframes ICP through the lens of Hierarchical Bayesian Modeling (HBM). Instead of testing conditional independence across subsets, BHIP models the variability of predictor effects across environments directly.

Core Probabilistic Model

BHIP assumes a Structural Causal Model (SCM) where the target $Y$ depends on a set of predictors $X$. The model posits:

1. Local Parameters: Each environment $e$ has specific regression coefficients $\beta^e$ describing the effect of predictors on $Y$.
2. Global Parameters: These local coefficients are drawn from a common global distribution governed by hyperparameters $\phi$ (e.g., a global mean $\mu$ and scale $\tau$).
   • $\beta^e_d \sim P(\beta^e_d | \phi_d)$
   • $\phi_d \sim P(\phi_d)$
3. Invariance Definition: A predictor is considered invariant (and thus causal) if:
   • Its effect is non-zero (the coefficient is credibly different from zero).
   • Its effect is stable across environments (the local coefficients $\beta^e_d$ are "close" to the global mean $\mu_d$).

Inference and Decision Rules

BHIP utilizes Markov Chain Monte Carlo (MCMC) to obtain posterior distributions for parameters. It employs two specific statistical tests to classify predictors:

1. HDI + ROPE (Region of Practical Equivalence):
   • Determines if an effect is non-zero.
   • Checks if the Highest Density Interval (HDI) of the posterior distribution lies substantially outside a small interval around zero (ROPE).
2. Pooling Factor ($\gamma_d$):
   • Quantifies the degree of information sharing (pooling) across environments.
   • Defined as $1 - \frac{\text{Variance of deviations}}{\text{Expected total variance}}$.
   • A value close to 1 indicates strong invariance (low heterogeneity in effects), while values closer to 0 indicate high heterogeneity.
   • Theorem 1: Under regularity conditions, the pooling factor converges to 1 for true causal parents as sample size and number of environments increase.

Decision Rule: A predictor is identified as an invariant causal parent if it passes both the HDI+ROPE test (non-zero effect) and the pooling factor test (high invariance, e.g., $\gamma_d > 0.85$).

3. Key Contributions

1. Unified Probabilistic Framework: BHIP synthesizes Hierarchical Bayes with Invariant Causal Prediction, offering a unified approach to test causal invariance directly within the model's posterior analysis.
2. Computational Scalability: By modeling parameter relationships rather than exhaustively testing subsets, BHIP avoids the $O(2^d)$ complexity of ICP. It scales linearly or polynomially with the number of predictors, making it viable for high-dimensional problems.
3. Uncertainty Quantification & Flexibility: Unlike ICP, BHIP provides full posterior distributions, allowing for rigorous uncertainty quantification. It also naturally incorporates prior information (e.g., sparsity-inducing priors like Horseshoe or Spike-and-Slab) to improve variable selection.
4. Theoretical Guarantees: The paper provides asymptotic guarantees (via the Bernstein-von Mises theorem) showing that the pooling factor converges to 1 for true invariant parents and 0 for non-invariant ones.

4. Experimental Results

The authors evaluated BHIP on synthetic and real-world datasets, comparing it against ICP, Hidden-ICP, EILLS, and a concurrent Bayesian method (BIP/VI-BIP).

• Bus Dwelling Problem (Synthetic):
  • BHIP successfully identified true causal predictors (boarding/alighting passengers) while excluding non-causal ones (traffic, time).
  • Result: BHIP outperformed ICP, which failed to detect one of the true causal predictors ($X_4$) due to its conservative nature.
• Educational Attainment (Real-world):
  • Applied to a dataset of US students. BHIP identified "achievement score" and "father's education" as invariant.
  • Novelty: BHIP detected "low income" as a negative invariant predictor, a relationship missed by ICP. It provided richer insights into environment-specific effects.
• Computational Complexity:
  • Experiments showed ICP runtime grows exponentially with the number of predictors, whereas BHIP's runtime grows much slower, confirming its scalability.
• Gene Perturbation Benchmark:
  • On yeast gene data, BHIP recovered core validated regulatory relationships, performing comparably to state-of-the-art Bayesian methods (BIP/VI-BIP) while maintaining high coverage (low Type I error).
• Synthetic Benchmarks:
  • BHIP achieved near-perfect exact discovery rates as the number of environments increased, significantly outperforming ICP in recall (power) while maintaining perfect precision.

5. Significance and Conclusion

BHIP represents a significant advancement in causal discovery by bridging the gap between frequentist invariance testing and Bayesian hierarchical modeling.

• Practical Impact: It solves the scalability bottleneck of ICP, enabling causal inference in high-dimensional settings (e.g., genomics, complex systems).
• Robustness: By leveraging partial pooling, it effectively handles data scarcity in individual environments by borrowing strength from the global structure.
• Interpretability: The probabilistic output allows researchers to make nuanced decisions based on the strength of evidence rather than binary "pass/fail" thresholds.

Limitations: The method relies on MCMC, which incurs higher computational overhead than simple frequentist tests (though less than exhaustive ICP). It also assumes the defined environments capture meaningful heterogeneity reflective of the causal structure.

In summary, BHIP offers a principled, scalable, and flexible alternative to ICP for discovering robust causal relationships in heterogeneous data, with particular advantages in high-dimensional settings and scenarios requiring uncertainty quantification.