🤖 machine learning

One Size does not Fit All: Heterogeneous Latent Space Alignment for Unsupervised Domain Adaptation

This paper proposes ADualVUOT, a novel unsupervised domain adaptation framework for medical image segmentation that combines a dual-encoder VAE with Continuous Normalizing Flows and Unbalanced Optimal Transport via Gaussian-Gromov-Wasserstein distance to effectively address domain shifts through enhanced latent space expressiveness and adversarial augmentation.

Original authors: Evangelia Koskinioti, Yi Shen, Georgios Stamou, Michael M. Zavlanos

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Evangelia Koskinioti, Yi Shen, Georgios Stamou, Michael M. Zavlanos

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are training a brilliant student (the AI model) to identify specific organs in medical scans. You give them a massive library of textbooks from one specific hospital (the Source Domain) where the lighting is perfect, the machines are new, and the patients are all from the same neighborhood. The student aced the test using these books.

However, when you send this student to a different hospital (the Target Domain) to do the real job, things go wrong. The new hospital uses older machines, the lighting is dimmer, and the patients have different body types. The student, who memorized the "look" of the first hospital's scans, gets confused and fails. This is the problem of Domain Shift.

The paper you provided, titled "One Size does not Fit All," proposes a new way to fix this. Here is the breakdown of their solution using simple analogies:

1. The Problem: "One Size Does Not Fit All"

Most previous methods tried to force the student to learn a single, rigid set of rules that applied to both hospitals equally. They tried to squeeze the complex, high-quality data from the first hospital and the messy, smaller data from the second hospital into the exact same "mental box" (a shared latent space).

The authors argue this is a bad idea. It's like trying to force a large, detailed map of a city and a small, sketchy map of a village into the same tiny pocket. You either lose the details of the city or you can't fit the village map at all.

2. The Solution: ADualVUOT

The authors built a new system called ADualVUOT. Think of it as a smart translation and training system with three main tricks:

Trick A: Two Different "Mental Boxes" (Dual-Encoder VAE)

Instead of forcing the student to use one mental box for both hospitals, they give the student two different boxes.

Box 1 (Source): A large, spacious box for the complex, detailed data from the first hospital.
Box 2 (Target): A smaller, more compact box for the limited data from the second hospital.

This allows the student to keep all the rich details of the training data without squishing it, while still learning how to translate that knowledge to the smaller, simpler target data.

Trick B: The "Shape-Shifting" Translator (CNF & GGW)

Even with two different boxes, how do you know if a "heart" in the big box matches a "heart" in the small box?

The Flow: They use a "shape-shifting" tool (Continuous Normalizing Flows) that stretches and twists the data inside the boxes to make it more expressive, like molding clay into the perfect shape.
The Translator: They use a special math tool called Gaussian Gromov-Wasserstein (GGW). Imagine you have two different languages. Instead of translating word-for-word (which fails if the languages are too different), this tool compares the structure of the sentences. It asks: "Does the relationship between these organs in the big box look similar to the relationship between organs in the small box?" This allows them to align the two different-sized boxes perfectly without forcing them to be the same size.

Trick C: The "Stress Test" Coach (Adversarial Augmentation)

To make sure the student is truly ready for the new hospital, the authors add a "stress test" coach.

Instead of just showing the student standard practice scans, this coach dynamically creates worst-case scenarios. It mixes the "style" of the first hospital with the second, adds random noise, and creates confusing images on the fly.
The coach tries to trick the student (by making the images harder to read), and the student tries to fight back and still get the answer right. This "adversarial" training makes the student incredibly robust, so when they face the real, messy data at the new hospital, they don't panic.

3. The Results

The authors tested this system on real medical data:

Prostate MRI: Scans from different hospitals with different machines.
Heart Scans: Switching between MRI and CT scans (which look very different).

In both cases, their new system (ADualVUOT) outperformed all previous methods. It was better at handling the differences between the "training hospital" and the "working hospital," proving that giving the AI flexible, different-sized mental boxes and stress-testing it with dynamic challenges works much better than trying to force everything into a single, rigid mold.

In short: The paper says that to fix AI when moving from one environment to another, stop trying to force a "one size fits all" solution. Instead, give the AI flexible tools to handle different data sizes and train it with a coach that creates the hardest possible practice scenarios.

Technical Summary: ADualVUOT for Unsupervised Domain Adaptation

Problem Statement

The reliable deployment of machine learning models in high-stakes environments, particularly medical imaging, is hindered by domain shift. Differences between training (source) and deployment (target) data distributions—caused by varying imaging devices, patient populations, or acquisition protocols—lead to significant performance degradation. While Unsupervised Domain Adaptation (UDA) aims to mitigate this by transferring knowledge from labeled source data to unlabeled target data, existing approaches face two primary limitations:

Restrictive Latent Representations: Many Optimal Transport (OT)-based methods enforce alignment in a shared latent space with identical dimensionality. This often forces the compression of diverse, high-dimensional source domains or the expansion of smaller target domains, discarding domain-specific information critical for downstream tasks.
Static Augmentation: Current data augmentation strategies often rely on handcrafted, predefined transformations or require prior knowledge of the target domain, which is unrealistic when target data is limited, partially observed, or evolving.

Methodology: ADualVUOT

The authors propose ADualVUOT (Adversarial Dual Variational Unbalanced Optimal Transport), a novel deep learning architecture designed specifically for medical image segmentation. The framework integrates four key components to address the limitations of shared latent spaces and static augmentation:

1. Dual-Encoder Variational Autoencoder with Continuous Normalizing Flows (CNF)

Instead of forcing source and target data into a single shared latent space, ADualVUOT employs a dual-encoder architecture:

Shared Encoder: A shared backbone extracts feature representations from input images.
Domain-Specific Heads: The features are split into domain-specific fully connected heads that predict parameters for two distinct Gaussian posteriors. This allows the source and target domains to be mapped into latent spaces of different dimensionalities ( $K$ for source, $P$ for target), accommodating the fact that source domains are often larger and more diverse than target domains.
Continuous Normalizing Flows (CNF): To overcome the restrictive inductive bias of Gaussian assumptions, the initial latent variables are transformed via CNFs. This creates more expressive, non-Gaussian posterior distributions, enhancing modeling flexibility.

2. Hierarchical Unbalanced Optimal Transport (UOT) with Gaussian Gromov-Wasserstein (GGW)

To align the heterogeneous latent spaces without requiring shared dimensionality, the framework utilizes a hierarchical OT formulation:

Embedding-Level Alignment (GGW): The Gaussian Gromov-Wasserstein distance is used to compare the internal geometric structures of the source and target latent distributions. This allows for transport between spaces of different dimensions by comparing eigenvalues of their covariance matrices rather than relying on pointwise distances.
Mini-Batch Level Alignment (UOT): An Unbalanced Optimal Transport plan is computed to establish soft correspondences between source and target samples within a mini-batch. This handles mass relaxation, accommodating potential distribution mismatches.
Semantic Consistency: The transport cost combines the geometric GGW distance with a semantic component based on the Dice Similarity Coefficient (DSC) between source labels and target pseudo-labels, ensuring that geometrically similar but semantically mismatched embeddings are not coupled.

3. Adversarial Data Augmentation (MaxStyle)

To improve robustness without relying on predefined augmentation policies, the authors adapt the MaxStyle strategy:

Style Mixing: Feature maps from source and target reconstruction decoders are mixed using linear combinations of their channel-wise means and standard deviations.
Adversarial Optimization: The mixing coefficients and additive noise parameters are optimized via gradient ascent on the source segmentation loss (while keeping network weights frozen). This process dynamically generates "worst-case" style compositions that challenge the model, forcing it to learn domain-invariant features without requiring prior knowledge of the target domain.

4. Training Objective

The total loss function combines:

Domain-specific Evidence Lower Bound (ELBO) losses for reconstruction and segmentation (for the source).
The UOT transfer loss to minimize domain discrepancy.
An adversarial loss term that minimizes the objective on the worst-case augmented samples.

Key Contributions

The paper claims the following contributions:

Novel Framework: The proposal of ADualVUOT, which leverages Optimal Transport theory via the Gaussian Gromov-Wasserstein distance to perform geometry-aware alignment between heterogeneous latent spaces (different dimensions).
Dual-Encoder CNF-VAE: The introduction of a dual-encoder architecture that projects source and target data into distinct latent spaces. The authors note this is the first work to integrate a dual-encoder architecture with OT-based latent alignment for UDA, avoiding restrictive shared-dimensionality assumptions.
Adversarial Augmentation: A dynamic augmentation strategy that generates challenging samples to improve robustness and facilitate alignment, eliminating the need for prior target domain knowledge or static policies.
Empirical Validation: Demonstration of effectiveness on multiple medical image segmentation benchmarks, showing consistent improvements over existing OT-based methods.

Experimental Results

The authors evaluated ADualVUOT on two medical imaging benchmarks:

Multi-Site Prostate MRI Dataset: Involving adaptation across seven different acquisition sites. ADualVUOT achieved the highest Dice scores across all target domains compared to baselines like PUFT, JUMBOT, and source-only models. Notable gains were observed on challenging sites (e.g., up to 8% improvement over PUFT on Site D).
Multi-Modality Whole Heart Segmentation (MM-WHS): Involving cross-modality adaptation (MRI $\leftrightarrow$ CT). ADualVUOT outperformed all baselines, with particularly significant improvements (up to 5% overall) in the challenging CT-to-MRI adaptation task.

Ablation Studies:

DualVUOT vs. ADualVUOT: Removing the adversarial augmentation (DualVUOT) resulted in lower performance, confirming the benefit of dynamic worst-case sample generation.
VUGW vs. DualVUOT: Replacing the hierarchical UOT-GGW with a standard Unbalanced Gromov-Wasserstein plan (VUGW) showed inferior performance, highlighting the importance of the hierarchical alignment and dual-encoder design.

Significance and Claims

The paper positions ADualVUOT as a solution to the "one size fits all" limitation in current domain adaptation methods. By decoupling latent representations and utilizing the GGW distance, the method preserves domain-specific information that is often lost in shared-space approaches. The integration of adversarial augmentation further enhances robustness against complex domain shifts common in medical imaging, such as variations in scanners and acquisition protocols. The authors claim that their approach achieves state-of-the-art results on standard medical benchmarks, with each component (dual-encoder, GGW alignment, and adversarial augmentation) contributing substantively to the overall performance improvement.