🤖 machine learning

Joint Causal Structure and Cluster Discovery Using Variational Inference

This paper proposes a novel variational inference framework that simultaneously infers latent variable clusters and their causal structures, addressing the limitation of existing methods that require pre-defined groupings, and demonstrates its effectiveness on both synthetic and real-world datasets.

Original authors: Avni Rajpal, Anubhav Kumar, Rishabh Karnad, Mohammad Emtiyaz Khan, P. K. Srijith

Published 2026-08-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Avni Rajpal, Anubhav Kumar, Rishabh Karnad, Mohammad Emtiyaz Khan, P. K. Srijith

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast landscape of data science, researchers often look for the hidden threads that connect different pieces of information. Imagine a room full of people talking; some conversations happen between individuals, while others occur within tight-knit groups. For decades, scientists have developed tools to map out who is talking to whom, treating every person as a single, independent voice. This approach, known as causal discovery, helps us understand how one thing causes another, moving beyond simple observation to uncover the true machinery of the world. However, in many complex systems, from the firing of neurons in a brain to the shifting patterns of global weather, the most important interactions do not happen between isolated individuals but between entire groups acting as a single unit. Until now, scientists have had to guess where these groups begin and end before they could study how the groups influence one another, a limitation that often obscured the bigger picture.

A team of researchers from the Indian Institute of Technology Hyderabad and the RIKEN Center for AI Project in Tokyo has developed a new method to solve this puzzle. They created a system that can simultaneously figure out how to group variables together and map out the causal connections between those groups, all without being told the groupings in advance. Instead of forcing the data into pre-defined boxes, their approach lets the data reveal its own natural clusters and the invisible lines of influence that run between them. By using a technique called variational inference, which allows the computer to learn from uncertainty rather than just seeking a single fixed answer, the researchers built a model that treats both the groupings and the connections as hidden secrets waiting to be uncovered.

The researchers tested their method on both computer-generated data and real-world datasets, including information about protein structures and climate patterns. In these tests, they compared their new approach against existing methods that assume the groups are already known or that rely on simpler, less flexible ways of guessing. The results showed that their method was significantly better at reconstructing the true underlying structure. For instance, when analyzing a dataset of proteins, their model correctly identified that the data formed two separate, unconnected groups with no causal links between them, whereas older methods incorrectly drew connections where none existed. Similarly, in climate data, the new approach successfully uncovered the non-linear relationships between different weather variables that other methods missed, accurately mapping both the clusters and the edges connecting them.

What makes this work particularly powerful is its ability to handle uncertainty. Rather than providing a single, rigid map, the system learns a probability distribution, meaning it can express how confident it is about each connection and each grouping. This is crucial for high-stakes fields like medicine or climate science, where knowing the limits of what we know is just as important as the facts themselves. The researchers found that their method worked best when it allowed for complex dependencies within the groups and between the edges, rather than assuming everything was independent. In the case of the climate data, a version of their model that accounted for these complex relationships achieved near-perfect accuracy, suggesting that the real-world mechanisms driving the data are indeed non-linear and intricate.

This study does not claim to have solved every problem in causal discovery, nor does it claim to work perfectly on every possible dataset. The researchers acknowledge that their current work is limited to specific types of data and that scaling the method to handle extremely large, high-dimensional datasets remains a challenge for the future. However, by demonstrating that it is possible to learn both the groups and the causal structures at the same time, they have opened a new path for understanding complex systems. Their work suggests that by letting the data speak for itself and by embracing the uncertainty inherent in real-world observations, we can build a clearer, more accurate picture of how the world works, one cluster at a time.

Technical Summary: Joint Causal Structure and Cluster Discovery Using Variational Inference

Problem Statement

Causal discovery traditionally focuses on inferring relationships between individual random variables, typically represented as a Directed Acyclic Graph (DAG). However, in complex domains such as brain imaging and climate modeling, interactions often occur among groups of variables rather than isolated units. While existing methods for "Cluster DAGs" (C-DAGs) or "Group DAGs" exist, they generally assume that the grouping of variables into clusters is known a priori through domain knowledge.

This paper addresses the gap where both the clusters (partitions of variables) and the causal structure (the DAG over these clusters) are latent and must be inferred simultaneously from observational data. The authors consider Vector-Valued Structural Causal Models (vSCMs), which generalize classical SCMs to handle vector-valued outputs generated by clusters of variables, accommodating both linear Gaussian and non-linear additive noise scenarios. While Niu et al. (2022) previously proposed a score-based approach for this joint estimation, this work introduces a novel variational inference framework to achieve the same goal, offering a Bayesian treatment for uncertainty quantification.

Methodology

The proposed approach utilizes Variational Inference (VI) to learn an approximate posterior distribution over the latent cluster assignments ( $C$ ) and the cluster graph structure ( $E_C$ ). The core of the method involves defining a generative model and optimizing an Evidence Lower Bound (ELBO).

1. Generative Model

The data $D$ is assumed to be generated by a vSCM where each cluster $X_{C_i}$ is a function of its parent clusters $Pa(X_{C_i})$ and a noise vector $\epsilon_i$ :
$X_{C_i} := f_{C_i}(Pa(X_{C_i}), \epsilon_i)$
The joint distribution factorizes according to the C-DAG structure. The authors consider two specific cases:

Linear Gaussian vSCM: The function $f$ is linear, and noise is multivariate Gaussian. The marginal likelihood can be computed in closed form by integrating out parameters.
Non-linear Additive Noise Model: The function $f$ is modeled as a neural network. Here, marginalization over parameters is intractable, so point estimation of parameters is used alongside variational inference over the structure.

2. Variational Approximation

Since the space of possible C-DAGs grows exponentially, exact Bayesian inference is infeasible. The authors approximate the true posterior $P(C, E_C | D)$ using a variational distribution $Q_\phi(C, E_C)$ that factorizes into distributions over clusters and edges:
$Q_\phi(C, E_C) = Q_T(C) Q_Z(E_C)$

Variational Families for Clusters ( $C$ ):

Factorized Model: Assumes independence between variable assignments, modeled as a product of categorical distributions.
Linear Autoregressive Model: Captures dependencies between cluster assignments by modeling the posterior as a product of conditionals, where the logits for the $m$ -th variable depend on previous assignments via a linear autoregressive process.

Variational Families for Graphs ( $E_C$ ):

Independent Binary Model: Assumes edges are independent Bernoulli variables.
Linear Autoregressive Model: Models edges as conditionally dependent on previous edges.
Conditional Model: Explicitly models the dependence of edges on the global cluster assignments ( $C$ ).
Autoregressive Conditional Model: Combines autoregressive edge dependencies with dependence on cluster assignments.

3. Optimization

The parameters of the variational distributions ( $T$ for clusters, $Z$ for edges) and model parameters ( $\Theta$ for non-linear cases) are learned by maximizing the ELBO.

Gradient Estimation: Since $C$ and $E_C$ are discrete, the authors use the REINFORCE score function estimator to approximate gradients. To mitigate high variance, an exponential moving average is used as a baseline.
Training: The optimization is performed using gradient-based methods (e.g., Adam, RMSprop) with multiple random restarts to avoid local optima.

Key Contributions

Joint Latent Inference via Variational Inference: The paper proposes a novel variational inference framework capable of simultaneously inferring latent variable clusters and the causal DAG over those clusters without prior knowledge of the grouping. This complements existing score-based approaches (e.g., Niu et al., 2022) by providing a full Bayesian posterior rather than point estimates.
Expressive Variational Families: The authors introduce and evaluate multiple variational families, specifically highlighting the importance of autoregressive and conditional models. These allow the model to capture complex dependencies between cluster assignments and between graph edges, which simpler factorized models miss.
Extension to Non-Linear vSCMs: The framework is extended to handle non-linear causal mechanisms using neural networks, allowing for the discovery of clusters and structures in complex, non-linear data-generating processes.
New Evaluation Metric: The authors introduce the Expected Expanded-Graph Structural Hamming Distance (ExG-SHD). Unlike standard SHD, this metric accounts for the approximation error in both the cluster assignments and the graph structure by expanding the C-DAG into a full variable-level DAG before computing the distance.

Experimental Results

The method was evaluated on synthetic and real-world datasets, comparing performance against a baseline score-based approach (Niu et al., 2022) using bootstrapping.

Synthetic Data (Linear): The autoregressive cluster model consistently outperformed factorized variants and the baseline. For instance, on a 4-variable "fork" dataset, the autoregressive model achieved an ExG-SHD of 0.02 and an Expected Rand Index (E-RI) of 0.99, nearly perfectly recovering the ground truth.
Synthetic Data (Non-linear): Neural network-based formulations significantly outperformed the baseline. Several variants achieved a perfect ExG-SHD of 0.00 on 4-variable datasets. The results indicated that capturing topological dependencies in edges (via autoregressive or conditional models) was crucial for structural accuracy.
Real-World Data:
- Protein Dataset: The neural models achieved near-perfect ExG-SHD and E-SHD scores (0.0), correctly identifying two unconnected clusters, whereas the baseline had a much higher error (ExG-SHD = 3.0).
- Climate Dataset: The neural conditional edge model achieved an ExG-SHD of 0.04 and a perfect E-RI of 1.0, suggesting the underlying causal mechanisms are inherently non-linear and that conditional modeling provides significant improvements over other methods.

Significance and Claims

The paper claims that its Bayesian treatment of learning distributions over clusters and structures offers distinct advantages over point-estimate methods:

Uncertainty Quantification: By learning a posterior distribution, the method provides a measure of uncertainty over the inferred C-DAGs, which is critical for high-risk applications.
Active Interventions: The probabilistic framework facilitates active intervention strategies to better understand the underlying causal model.
Handling Latent Confounding: The approach addresses scenarios where independent noise assumptions are violated by grouping variables, effectively handling latent confounding between exogenous variables.

The authors conclude that while their current approach is effective for low-to-moderate dimensional datasets, future work aims to scale the method to high-dimensional data. They emphasize that the ability to jointly discover latent clusters and causal structures without domain priors represents a significant step forward in causal discovery for complex systems.