Disentanglement of Variations with Multimodal Generative Modeling

Technical Summary: Information-Disentangled Multimodal VAE (IDMVAE)

Problem Statement
Multimodal data, such as video with audio or images with captions, is prevalent across domains. A core challenge in learning from such data is integrating information across views to create universal, transferable representations. While Variational Autoencoders (VAEs) have emerged as a powerful paradigm for this, recent approaches recognize that real-world datasets contain both shared information (common across modalities) and private information (modality-specific).

Existing methods attempt to separate these components using two latent variables: $z$ for shared information and $w_m$ for private information in modality $m$. However, these methods struggle to achieve maximal disentanglement, particularly on challenging datasets where the likelihood model is insufficient. Without clean separation, shared information leaks into private encodings and vice versa, resulting in weak cross-modal coherence, wasted model capacity, and inadequate generative quality. The central question addressed is: How can we achieve maximal disentanglement between shared and private variables so that learned representations are complete and non-redundant?

Methodology
The authors propose Information-disentangled Multimodal VAE (IDMVAE), a framework that combines rigorous mutual information (MI) regularizations with diffusion-based priors. The model assumes that for each modality $x_m$, the data is generated jointly by a shared latent variable $z$ and a private latent variable $w_m$, with independent priors.

The method consists of three primary components:

1. Likelihood Modeling with Multimodal VAE:
   The foundation is built upon the MMVAE+ model, which utilizes a Mixture-of-Experts (MoE) scheme to aggregate view-specific posteriors into a global posterior for $z$. To prevent the decoder from taking a "shortcut" by using leaked shared information during cross-view reconstruction, MMVAE+ employs an auxiliary prior for the private variable of the target view. This establishes a baseline Evidence Lower Bound (ELBO) for reconstruction.

2. Shared Variable Extraction via Cross-View MI Maximization:
   To explicitly enforce the extraction of shared factors, the authors introduce a regularization term, $L_{CrossMI}$. While likelihood maximization alone does not guarantee disentanglement, maximizing the mutual information between shared representations of different views ($I(z_m; z_n)$) serves as a lower bound for $I(z_m; x_n)$. This is achieved using a contrastive estimate of mutual information (similar to InfoNCE), encouraging $z_m$ and $z_n$ to be highly correlated while minimizing redundancy with private variables.

3. Disentanglement via Generative Augmentation:
   To remove redundancy between $z$ and $w_m$ (ensuring $w_m$ does not retain shared information), the authors propose a cycle-consistency style loss, $L_{GenAug}$. This method synthesizes new samples $x^+_m$ by combining the shared posterior $z$ from one sample with the private posterior $w'_m$ from another sample. The model is then regularized to ensure that when $x^+_m$ is re-encoded, the resulting posterior $q(w_m|x^+_m)$ matches the original private posterior $q(w_m|x'_m)$. Crucially, these augmentations are generated by the model itself, requiring no domain-specific knowledge or external data augmentation strategies.

4. Diffusion Priors:
   To overcome the limitations of simple Gaussian priors, which may not capture the complex structure of multimodal latent spaces, the authors parameterize the prior $p(z)$ using a diffusion model. Instead of a fixed distribution, $p(z)$ is modeled as a denoising process. This allows the model to learn a richer, more structured latent space, improving the capacity for representation learning and generation.

The final objective function combines the MMVAE+ ELBO, the cross-view MI loss, the generative augmentation loss, and the diffusion prior loss.

Key Contributions

• MI-Based Regularization: Unlike prior works relying on capacity-based heuristics or shortcut-preventing mechanisms, IDMVAE employs mutual information-based regularizations to ensure disentanglement.
• Model-Generated Augmentation: The paper introduces a cycle-consistency loss using generative augmentations to remove redundancy between shared and private latents without requiring external domain knowledge.
• Diffusion Priors: The integration of diffusion models into the latent prior space enhances the representational capacity of the VAE, moving beyond simple Gaussian assumptions.
• Synergy: The paper demonstrates that MI-based objectives and diffusion priors are complementary, jointly overcoming the limitations of pure likelihood modeling.

Experimental Results
The authors evaluated IDMVAE on three datasets:

1. PolyMNIST-Quadrant: A modified MNIST dataset where digits (shared) are placed in different quadrants (private) with varying backgrounds. IDMVAE achieved superior latent classification accuracy (0.983 for shared digits vs. 0.382 for MMVAE+) and significantly improved cross-modal generation coherence.
2. CUB-200-2011: A fine-grained bird classification dataset with image and text modalities. IDMVAE outperformed baselines in extracting shared category information and private directional information, demonstrating better cross-modality generation (e.g., text-to-image) with higher semantic coherence.
3. TCGA (The Cancer Genome Atlas): A multi-omics dataset for predicting patient mortality. IDMVAE showed improved prediction accuracy (0.718) compared to baselines, indicating that clean disentanglement separates predictive information effectively between shared and private latent spaces.

In all experiments, ablation studies confirmed that removing either the cross-view MI loss or the generative augmentation loss degraded performance, validating the necessity of both components. The addition of diffusion priors consistently provided further gains in generation quality and coherence.

Significance and Claims
The paper claims that IDMVAE demonstrates a "clean separation" between shared and private information, leading to superior generation quality and semantic coherence on challenging datasets. The authors argue that their approach addresses the insufficiency of likelihood models in existing methods by explicitly enforcing disentanglement through information-theoretic regularizations.

The significance of the work lies in its ability to learn robust, transferrable representations that are both complete (capturing all relevant information) and non-redundant. By combining generative modeling with information bottleneck principles and diffusion priors, the method offers a more powerful paradigm for unsupervised multimodal representation learning than existing state-of-the-art VAEs or non-generative disentanglement methods. The authors modestly note limitations, such as the difficulty in generating high-fidelity images on the CUB dataset due to data volume constraints, and suggest future work on handling missing modalities and integrating pre-trained diffusion models in the input space.