Technical Summary: CaSPECT – Discovering Causally Homogeneous Subgroups via Directed Spectral Clustering

1. Problem Statement

Traditional causal inference often focuses on estimating the Average Causal Effect (ACE) or the Conditional Average Causal Effect (CACE) across the entire population or based on observed covariate similarity. However, these approaches can obscure treatment effect heterogeneity, where interventions yield beneficial results for some subgroups while harming others. Existing unsupervised causal clustering methods (e.g., causal k-means, hierarchical clustering) typically group individuals based on estimated treatment effect functions but often rely on classical clustering techniques that ignore the underlying causal topology. Furthermore, many methods require pre-specified propensity score models or struggle when subpopulations overlap in marginal covariate distributions but differ in causal mechanisms.

The paper addresses the challenge of discovering causally homogeneous subgroups from observational data without relying on covariate similarity or pre-specified parametric models. The goal is to uncover latent structures where individuals share similar causal propagation pathways, rather than just similar observed features.

2. Methodology: The CaSPECT Framework

CaSPECT (Causal Spectral Clustering) is a four-stage pipeline that shifts the clustering objective from feature space to causal topology. It constructs a similarity graph based on the structure of a learned Directed Acyclic Graph (DAG) and performs spectral clustering on this causal structure.

Stage 1: DAG Formation and Orientation

The framework begins by recovering a fully oriented DAG ($G^*$) from observational data:

1. Skeleton Recovery: A bootstrap-stabilized PC algorithm is used to identify a stable causal skeleton. Edges are retained only if their inclusion frequency ($f_{uv}$) across $B$ bootstrap resamples exceeds a threshold $\theta$, bounding false positives.
2. Orientation Validation Score (OVS): Since the PC algorithm cannot orient all edges (due to Markov equivalence), CaSPECT introduces a novel OVS to resolve ambiguities. The OVS combines:
   • Bootstrap Evidence ($\delta_{PC}$): Net orientation evidence from the PC algorithm ($f_{uv} \cdot (\rho_{uv} - \rho_{vu})$).
   • LiNGAM Evidence ($\delta_{L}$): Directional preference from DirectLiNGAM based on non-Gaussianity of error terms.
   • The score is weighted by $w_{PC}$ and $w_L$ (where $w_L$ scales with the number of non-Gaussian variables).
3. Resolution Hierarchy: Edges with $|OVS| \le \gamma$ remain ambiguous. They are resolved via an acyclicity filter, Meek's rules, domain/temporal ordering, or edge contraction (merging nodes to preserve DAG acyclicity rather than symmetrizing edges, which would violate the backdoor criterion).

Stage 2: Causal Edge Weight Estimation

Once the DAG is oriented, edges are weighted by the magnitude of the Average Causal Effect (ACE) identified via the backdoor adjustment set:

• Linear Edges: If a RESET test confirms linearity, Ordinary Least Squares (OLS) estimates the structural coefficient $\beta_{uv}$.
• Nonlinear Edges: If nonlinearity is detected, Double Machine Learning (DML) with cross-fitting is used to estimate the ACE, providing robustness against nuisance function estimation errors.
• Stability Weighting: Final edge weights ($A^{stab}_{uv}$) are the product of the estimated ACE magnitude and the bootstrap inclusion frequency ($f_{uv}$). This attenuates the contribution of unstable edges while preserving strong causal signals.

Stage 3: Chung Laplacian Construction and Spectral Embedding

The weighted adjacency matrix is normalized to form a transition matrix $P$. CaSPECT utilizes Chung's Directed Laplacian ($L = I - \frac{1}{2}(P + P^*)$), where $P^*$ is the time-reversed matrix. This operator is symmetric and positive semi-definite, allowing for spectral decomposition.

• The embedding dimension $K^*$ is selected based on the largest spectral gap (Cheeger's inequality), corresponding to well-separated causal communities.
• The data is projected onto the eigenvectors of $L$ to obtain a low-dimensional spectral embedding ($\tilde{X}$) where proximity reflects similarity in causal propagation pathways.

Stage 4: Clustering

Standard k-means clustering is applied to the spectral embedding $\tilde{X}$. The optimal number of clusters $k$ is selected using the Silhouette score and the Gap statistic.

3. Key Contributions

• Causal Spectral Clustering: Proposes a framework that clusters individuals based on the topology of a learned DAG rather than observed covariate space, addressing the limitation of methods that fail when subpopulations overlap in marginal distributions.
• Orientation Validation Score (OVS): Introduces a hybrid mechanism combining PC bootstrap stability and LiNGAM non-Gaussianity to robustly orient edges, with theoretical guarantees of consistency under standard assumptions.
• Edge Contraction vs. Symmetrization: Advocates for merging ambiguous nodes (contraction) rather than symmetrizing edges, which preserves the acyclicity required for valid backdoor adjustment and causal interpretation.
• Theoretical Consistency: Establishes the almost-sure consistency of the full pipeline, demonstrating that as sample size $n \to \infty$, the stable skeleton recovers the true edge set, edge weights converge to true ACE magnitudes, and the spectral embedding error decays at a parametric rate ($O(n^{-1/2})$).
• Robustness to Confounding: Demonstrates the ability to correct for severe confounding without pre-specified propensity score models by identifying causally comparable subpopulations.

4. Results and Validation

Simulation Studies

The method was evaluated on synthetic data with known ground truth across three scenarios:

1. Linear DAG with Non-Gaussian Errors: CaSPECT showed improving causal effect estimation (decreasing RMSE) with sample size. However, clustering performance (Adjusted Rand Index) did not strictly improve with $n$, suggesting the spectral embedding captures global causal structure but may be less sensitive to specific treatment-intercept shifts defining subpopulations.
2. Mixed Linearity: The framework successfully adapted to nonlinear edges using DML. While RMSE increased slightly due to the variance of DML in finite samples, the method correctly identified the structural heterogeneity.
3. Latent Confounding: Under mild violations of causal sufficiency, the bootstrap stability weighting effectively suppressed spurious edges, allowing for improved structural recovery and reduced estimation bias as sample size increased.

• Ablation Studies: Confirmed that removing OVS (relying only on PC) significantly reduced orientation accuracy and clustering performance. Removing bootstrap stability weighting had a minor effect in large samples but highlighted its regularization role in finite samples. Symmetrizing edges (instead of contracting) destroyed directional information, leading to poor clustering.

Real-World Applications

• LaLonde CPS1 Dataset: CaSPECT successfully separated the sample into two clusters without a propensity score model.
  • Cluster 1: Economically stable, non-Black, married individuals (control units only).
  • Cluster 2: Economically disadvantaged, diverse individuals (containing all treated units).
  • Result: The global ACE was negative ($-1.70$), reflecting severe confounding. Restricting analysis to Cluster 2 revealed a positive, statistically significant ACE ($+1.40$), consistent with the experimental benchmark, demonstrating the method's ability to isolate causally comparable subgroups.
• IHDP Dataset: The pipeline separated treated and control infants into distinct clusters based on causal structure, despite high-dimensional covariate imbalance. It correctly identified that the all-control cluster (Cluster 1) had a higher ground-truth potential treatment effect, highlighting the identification limits (positivity violation) for that group.
• 401(k) Dataset: The method identified two economically distinct household subgroups.
  • Cluster 1: High-income, dual-earner households.
  • Cluster 2: Lower-income, single-earner households.
  • Result: Contrary to intuition, Cluster 2 exhibited a larger treatment effect ($9,873 vs $7,884). The authors interpret this as Cluster 1 households having existing savings channels (substitution effect), whereas 401(k) eligibility for Cluster 2 represented a genuine expansion of savings infrastructure.

5. Significance and Claims

The paper claims that CaSPECT provides a principled framework for mapping specific treatment responses to localized structural interventions. Its primary significance lies in:

• Decoupling from Covariate Similarity: It successfully distinguishes overlapping subpopulations where traditional CACE or mixture SEM methods fail due to marginal distribution overlap, by focusing on "how influence flows" rather than "what values variables take."
• Implicit Common Support Enforcement: By embedding individuals based on causal propagation pathways, the method naturally separates causally incomparable observations, effectively enforcing common support without explicit trimming or matching.
• Integration of Discovery and Estimation: It unifies causal discovery (DAG learning), effect estimation (DML/OLS), and clustering (Spectral methods) into a single consistent pipeline.

The authors acknowledge trade-offs: the method's performance is dependent on the accuracy of the initial DAG formation and the assumption of causal sufficiency. While bootstrap weighting mitigates minor uncertainty, fundamentally misspecified graphs can propagate errors. The paper positions CaSPECT not as a replacement for all causal methods, but as a complementary, data-driven approach for uncovering latent causal regimes that traditional methods overlook.