Discrete Causal Representations from Heterogeneous Domains: A Bayesian Approach with Social Survey Applications

Technical Summary: Bayesian Multi-Domain Causal Representation Learning

Problem Setting
Causal representation learning (CRL) aims to infer high-level latent causal concepts from observed low-level measurements. While identifiability of such representations has been extensively studied theoretically, practical, uncertainty-aware methods for real-world data remain underexplored. This work addresses the challenge of learning discrete causal representations from heterogeneous data collected across multiple domains (environments).

The core difficulty lies in the fact that causal variables are often latent and only indirectly observed through high-dimensional proxies (e.g., survey responses). Furthermore, distribution shifts across domains typically arise from sparse, localized changes in underlying causal mechanisms (interventions), while other parts of the generative process remain invariant. Standard causal discovery methods often fail in this setting due to the mismatch between observed data and latent variables, and the lack of perfect, single-node interventions in real-world scenarios.

Methodology
The authors propose a hierarchical Bayesian model that formalizes the data-generating process as a latent variable model with discrete causal concepts. The framework integrates three key components:

1. Generative Model Structure:

   • Latent Causal Model: The latent variables $Z = (Z_1, \dots, Z_L)$ are discrete and follow a causal graph $G$. The joint distribution factorizes into causal mechanisms $p(Z_\ell | PA_\ell)$.
   • Multi-Domain Interventions: Differences across domains are modeled as unknown, soft (imperfect), multi-node interventions on the latent causal mechanisms. The model assumes an invariant measurement process $p(X|Z)$ across all domains, while the latent distribution $p(Z)$ varies due to these interventions.
   • Discrete Latents: The model focuses on discrete latent variables, allowing the marginal domain-specific distributions $p(X)$ to be expressed as finite mixture models with domain-specific mixture weights but shared mixture components.

2. Bayesian Priors and Assumptions:

   • Sparse Mechanism Shifts: To handle finite samples and avoid overfitting, the authors refine the sparse mechanism shift hypothesis. They introduce $\epsilon$-intervention detectability (Assumption 3.3), requiring that any active intervention induces a distributional change (measured by KL divergence) of at least $\epsilon$.
   • KL-Thresholded Shift Prior: To operationalize detectability, the authors devise a novel prior for intervention shifts in logit space. This prior uses a quadratic approximation of the KL divergence to truncate shifts that are too small to be distinguishable from noise, effectively ruling out negligible interventions.
   • Sparsity Induction: A Beta-Bernoulli prior is placed on intervention targets to induce sparsity, reflecting the assumption that distribution changes affect only a few factors.
   • Measurement Model: An additive Gaussian measurement model is used, where latent concepts contribute additively to the mean of observed variables. This parameterization aids interpretability by treating latent levels as ordinal factors.

3. Inference Scheme:

   • Due to the highly multimodal nature of the posterior (arising from permutation symmetries in the latent ordering), standard Gibbs sampling is insufficient.
   • The authors employ Sequential Monte Carlo Sampling (SMCS) with likelihood tempering. This population-based method evolves a set of particles through a sequence of intermediate distributions (from the prior to the posterior) using reweighting, resampling, and mutation (Gibbs updates) steps.
   • The inference leverages conjugate updates for most parameters and utilizes Pólya-Gamma data augmentation to handle non-conjugate logit parameters efficiently.

Key Contributions

1. Conceptual: The paper formulates a hierarchical Bayesian model for interventional multi-domain data, translating causal assumptions (modularity, sparse shifts) into specific modeling choices and priors. It refines the sparse mechanism hypothesis for finite-sample scenarios via the $\epsilon$-detectability condition.
2. Technical: The authors develop a novel KL-divergence truncated prior for intervention shifts and an SMCS-based inference procedure capable of exploring complex, multimodal posteriors over latent causal structures.
3. Empirical: The work demonstrates the application of a complete unsupervised CRL pipeline—from model formulation to estimation and interpretation—on complex real-world data, specifically social and political survey datasets.

Results
The method was evaluated on synthetic data and two real-world case studies:

• Synthetic Experiments: The SMCS approach outperformed standard Gibbs sampling, Parallel Tempering, and Variational Inference in recovering the true latent causal structure and intervention targets. Ablation studies showed that removing the KL-thresholding or sparsity priors improved predictive fit (LPPD) but significantly degraded the recovery of the underlying causal structure, highlighting a trade-off between prediction and causal discovery.
• World Values Survey (WVS): Applied to cross-national survey data (2017–2022), the model recovered three interpretable latent factors: economic hardship, demographics, and cultural conservatism. These aligned with the established Inglehart–Welzel map of cross-cultural variation. The model inferred a causal structure where economic hardship and demographics influence cultural conservatism, and identified country-specific intervention patterns consistent with known socioeconomic shifts.
• US Political Opinion Data:
  • Real Data: Analysis of survey data from different US regions and urbanicity levels revealed a largely one-dimensional latent structure reflecting partisan divisions (rural vs. urban gradients), consistent with known political trends.
  • LLM-Generated Data: Using a "semi-synthetic" dataset generated by Large Language Models with controlled interventions, the model successfully recovered a five-variable causal graph involving Regulation, Partisanship, Trade, Immigration, and Poverty. The recovered dependencies (e.g., Regulation $\to$ Poverty) aligned with the "ground-truth" treatment effects embedded in the synthetic generation process.

Significance and Claims
The paper claims to be one of the first successful demonstrations of applying a complete unsupervised CRL pipeline to complex real-world data. Its significance lies in moving beyond idealized synthetic settings and theoretical identifiability results to develop a robust, uncertainty-aware Bayesian framework capable of handling:

• Finite sample sizes.
• Imperfect, multi-node, and soft interventions.
• Partial identifiability and model misspecification.

By targeting a posterior distribution rather than a point estimate, the method quantifies both epistemic uncertainty (due to limited data) and aleatoric uncertainty (due to inherent ambiguities in the generative process). The authors position this work as a step toward bridging the gap between theoretical causal representation learning and practical applications in fields like social science, where causal variables are latent and interventions are unobserved.