Technical Summary: Causal Variational Deep Embedding (CAUVADE)

Problem Statement

Deep generative models (e.g., VAEs, GANs, Diffusion models) are designed to reproduce the observational distribution of training data. However, when training data contains spurious correlations driven by unobserved confounders ($U$), these models inherit these biases. In a causal setting, if a user wishes to manipulate an attribute $X$ (e.g., digit identity) and observe the resulting distribution of a dependent attribute $Y$ (e.g., background color), standard models output the observational conditional $P(Y|X)$ rather than the interventional distribution $P_x(Y)$.

The core challenge is partial identifiability. When $U$ is hidden, the observational data $P(X, Y)$ is consistent with a set of distinct causal mechanisms, each predicting a different $P_x(Y)$. Existing causal generative approaches often resolve this ambiguity by imposing strong structural assumptions (e.g., specific functional forms or noise independence) to select a single interventional distribution. In image domains, such assumptions are rarely warranted; a model returning a single point estimate is overconfident, having silently chosen one element from a set of indistinguishable causal explanations. The goal is to characterize the feasible region—the set of all interventional distributions consistent with the observed data and defensible structural assumptions—rather than committing to a single point.

Methodology

1. Canonical Augmented Structural Causal Models (CASCMs)

The authors propose a theoretical foundation based on Augmented Structural Causal Models (ASCMs) extended to image generation. An ASCM includes an image variable $I$ generated by semantic variables ($X, Y, Z$) and exogenous noise.

• Theoretical Justification: The paper proves that the class of Canonical ASCMs (CASCMs) is dense in the class of all ASCMs compatible with a given causal diagram, under both observational and interventional 1-Wasserstein distance ($W_1$).
• Canonical Structure: In a CASCM, the unobserved confounder $U$ is restricted to a discrete latent variable with bounded support ($U \in \{1, \dots, d\}$), while continuous variation is absorbed into independent exogenous noises ($E_Z, E_X, E_Y, E_I$). This reduction allows the infinite-dimensional search space of general ASCMs to be approximated by a tractable class where the confounder is a discrete cluster.

2. CAUVADE Architecture

Causal Variational Deep Embedding (CAUVADE) instantiates the CASCM class as a Gaussian-mixture Variational Autoencoder (VAE).

• Latent Variables:
  • Cluster $C$: A discrete variable representing the canonical confounder $U$.
  • Continuous Latent $Z$: Bundles the pre-treatment attribute and independent exogenous noises.
• Generative Process: The model defines a joint distribution where $C$ and $Z$ drive the generation of $X$, $Y$, and the image $I$.
• Intervention: To generate interventional samples $P_x(I)$, the model samples $(C, Z)$ from their joint prior (not the posterior given $X$), fixes $X=x$, and propagates through the decoders for $Y$ and $I$. This breaks the confounding path, distinguishing intervention from selection.

3. Training and the $\gamma$-Sweep

CAUVADE is trained on confounded data where only $X$ is labeled. The objective function combines standard VAE terms with two specific regularizers:

1. Treatment-Consistency Term ($L_X$): An auxiliary classifier ensures the latent representation $(C, Z)$ contains sufficient information to predict the observed treatment $X$.
2. Cluster-Entropy Regularizer ($L_{ent}$): An entropy term on the cluster posterior $q_\phi(C|I)$ weighted by a scalar $\gamma$.
   • Low $\gamma$: The posterior collapses (low entropy), leading to a deterministic VAE-like behavior that tracks the observational conditional $P(Y|X)$.
   • High $\gamma$: The posterior becomes uniform (high entropy), decoupling the image from the specific confounder realization and averaging over the prior.
   • The Sweep: By sweeping $\gamma$ over a range $[0, \infty)$, CAUVADE traces a continuous family of generators. Each $\gamma$ corresponds to a different optimum that fits the observational data to comparable likelihood but induces a distinct interventional distribution $P_x(I)$.

Key Contributions

1. Canonical Augmented SCM: Introduction of a class of structural models where unobserved confounders are discrete latent clusters. The authors prove this class is dense in the space of all compatible ASCMs under Wasserstein distance, justifying the restriction to this simpler class for image domains.
2. CAUVADE Framework: A practical instantiation of the canonical class using a mixture VAE. It introduces an entropy regularizer on the cluster posterior to navigate the feasible region of interventional distributions without requiring strong structural assumptions or auxiliary supervision.
3. Partial Identification via $\gamma$-Sweep: Demonstration that varying the entropy weight $\gamma$ traces a curve through the feasible region of causal explanations. This allows the model to produce a diverse set of interventional samples consistent with the data, rather than a single biased estimate.

Experimental Results

The authors evaluate CAUVADE on three datasets: Confounded Color-MNIST, CelebA, and MIMIC-CXR-JPG.

• Color-MNIST: In a setting where digit identity and background color are spuriously correlated via a hidden confounder, standard VAEs and ANCMs collapse to single point estimates (either the confounded conditional or a single point within the Manski bound). CAUVADE, by sweeping $\gamma$, produces a diverse set of digit-color associations that span the entire feasible region defined by Manski's bounds.
• CelebA: Modeling the effect of "Young" on "Heavy Makeup" with "Attractive" as a hidden confounder, CAUVADE generates samples that visually align with the unconfounded ground truth across different $\gamma$ values, whereas baselines fail to disentangle the spurious correlation.
• MIMIC-CXR-JPG: Modeling the causal effect of Pneumonia on Lung Opacity (confounded by patient position), CAUVADE achieves a significantly lower Fréchet Inception Distance (FID) against an unconfounded reference (284.57) compared to VAE (354.97) and ANCM (407.73). The baselines either inherit the bias directly or commit to a single incorrect causal explanation.

Significance and Claims

The paper claims that in the presence of hidden confounders, the "right object to model" is the feasible region of interventional distributions, not a single point estimate.

• Modest Claims: The authors do not claim to identify the true causal mechanism (which is impossible without further assumptions). Instead, they claim to provide a method that faithfully characterizes the set of plausible mechanisms consistent with the data.
• Practical Impact: By avoiding the overconfidence of single-point estimators, CAUVADE offers a more honest representation of uncertainty in causal image generation. The method is presented as a practical tool for "deconfounded image generation" that produces diverse, high-quality samples covering the range of causal explanations allowed by the observational data.
• Theoretical Contribution: The density theorem (Theorem 2.2) provides a rigorous justification for using discrete latent clusters to approximate complex, continuous unobserved confounders in high-dimensional image spaces, bridging the gap between theoretical causal inference and deep generative modeling.