Identifiable Multimodal Causal Representation Learning under Partial Latent Sharing

Technical Summary: Identifiable Multimodal Causal Representation Learning under Partial Latent Sharing

Problem Statement
Causal Representation Learning (CRL) aims to uncover meaningful latent variables and their causal structures from high-dimensional observational data. However, a critical bottleneck in CRL is identifiability: without guarantees, learned representations may be structurally different from the true data-generating process, undermining interpretability and robustness.

Existing multimodal CRL approaches face significant limitations:

1. Restrictive Assumptions: Many methods assume bijective mixing functions (where observed and latent dimensions are equal), which is unrealistic for scenarios like image data where observations are high-dimensional and latent variables are low-dimensional (undercomplete setting).
2. Limited Sharing Patterns: Prior work often focuses on either non-overlapping latent structures or fully shared structures (where all modalities depend on the same set of latents). Few address partially shared structures, where specific subsets of modalities share latent variables while others remain modality-specific.
3. Dependency on Auxiliary Signals: Many methods rely on supervision, interventions, or temporal dependencies, which are not always available in real-world settings.

This paper addresses the challenge of establishing component-wise identifiability for causal latent representations in a multimodal setting with partially shared latent structures, without requiring parametric distribution assumptions, supervision, or interventions.

Methodology

1. Theoretical Framework
The authors formulate a latent variable model where observed modalities $x^{(m)}$ are generated from subsets of causal latent variables $z_{A_m}$ via smooth, injective (but not necessarily bijective) mixing functions $f_m$. The latent variables $z$ follow a structural causal model (SCM) defined by $z_j = g_j(Pa(z_j), \epsilon_j)$.

The core theoretical contribution is the proof of component-wise identifiability (recovering each latent variable up to a smooth invertible transformation) under the following conditions:

• Injectivity & Properness: Mixing functions are injective and proper (preventing information loss at extreme values).
• Causal Structural Sparsity: The causal graph of latent variables exhibits sparsity, specifically restricting causal edges between specific variables of different modalities.
• Partial Sharing: No single latent variable is shared across all modalities; sharing occurs only among subsets.
• Mixing Density: The causal structure ensures that non-permutation transformations would increase the density of causal edges, violating sparsity constraints.

These conditions allow the recovery of both shared variables (common to subsets of modalities) and modality-specific variables up to component-wise bijections, even in undercomplete scenarios ($|A_m| \leq d_m$).

2. Learning Framework
To instantiate these theoretical guarantees, the authors propose a generative framework comprising three main components:

• Autoencoder Structure: Modality-specific encoders map observed data $x^{(m)}$ to latent estimates $\hat{z}_{A_m}$, which are then decoded to reconstruct observations.
• Wasserstein-Based Pluggable Module: A novel, differentiable module learns the partially shared latent structure. It formulates the alignment of shared variables as an optimal transport problem. By minimizing a transport cost (based on Wasserstein distance) between latent variables of different modalities, the module learns a permutation matrix $P$ that identifies which latent components are shared across modalities. This module is architecture-agnostic and requires no auxiliary supervision.
• Causal Structure Learning: The framework employs a Masked Autoregressive Normalizing Flow to model exogenous noise and a learnable adjacency matrix to infer the causal graph. Sparsity ($L_1$ regularization) and acyclicity constraints are enforced to satisfy the theoretical identifiability conditions.

Key Contributions

1. Component-wise Identifiability Guarantees: The paper establishes that both shared and modality-specific latent variables can be identified component-wise in a multimodal setting with partial sharing, relaxing the strict bijectivity assumption required by prior work.
2. Undercomplete Setting: The theoretical results hold even when the number of observed variables exceeds the number of latent variables in each modality.
3. Architecture-Agnostic Module: Introduction of a differentiable Wasserstein-based module that recovers the shared structure permutation matrix without requiring additional training data, supervision, or interventions.
4. Generalization: The work generalizes previous block-wise identification results to a more flexible setting where modalities are not necessarily augmented versions of each other.

Experimental Results
The authors evaluate their approach on synthetic numerical datasets (2, 3, and 4 modalities), a synthetic 3D image-text dataset (MultiModal3DIdent), and a gene regulatory network dataset (SERGIO).

• Metrics: Performance is measured using Mean Correlation Coefficient (MCC), Coefficient of Determination ($R^2$), and Enhanced Structural Hamming Distance (EnSHD).
• Findings:
  • The proposed method outperforms state-of-the-art baselines (CausalVAE, Multimodal Contrastive Learning, MultiBio, MultiView) in terms of $R^2$ across all numerical datasets, indicating better recovery of the ground-truth latent structure.
  • On the MultiModal3DIdent dataset, the method achieves superior $R^2$ scores compared to baselines.
  • On the Gene Regulatory Network dataset, the method achieves the highest MCC (0.98), confirming applicability to real-world biological data.
  • The method demonstrates competitive or superior performance in recovering causal graphs (EnSHD) compared to baselines that rely on contrastive learning or block-wise identification.

Significance and Claims
The paper claims that its primary significance lies in providing weaker theoretical conditions for identifiability compared to existing works. By removing the need for bijective mixing functions, parametric distribution assumptions, and auxiliary signals (supervision/interventions), the proposed framework offers a more practical and flexible approach to learning interpretable causal representations.

The authors emphasize that their component-wise identifiability guarantees ensure that learned representations are not just structurally similar but individually recoverable, which is crucial for downstream tasks requiring specific causal factors. The introduction of the Wasserstein-based module provides a practical mechanism to handle the complex alignment of partially shared structures in a differentiable manner, bridging the gap between theoretical identifiability and deep learning implementation.