Technical Summary: Together, Then Apart (TTA) for Multimodal Survival Analysis

1. Problem Statement

Multimodal survival analysis aims to improve cancer prognosis by integrating heterogeneous biomedical data, specifically histopathology whole-slide images (WSIs) and genomic profiles. While current state-of-the-art methods often rely on aligning these modalities into a shared latent space to capture cross-modal correlations, the authors identify a critical limitation: over-alignment collapse.

Aggressive alignment forces heterogeneous signals (e.g., spatial morphological patterns in WSIs and sparse gene-level variations) into a single representation prematurely. This process can suppress modality-specific evidence that is critical for survival prediction. Empirical observations indicate that such over-alignment can cause multimodal models to underperform strong single-modality baselines (particularly WSI-only models) by diluting distinctive signals and reducing representational diversity. Existing methods that attempt to disentangle shared and specific representations often treat these objectives independently, lacking a unified principle to jointly encourage semantic agreement while preserving modality identity.

2. Methodology: The TTA Framework

The authors propose Together, Then Apart (TTA), a representation learning framework based on the principle that effective models must first discover shared patterns across modalities and then preserve modality-specific signals. The framework consists of four main components:

2.1 Tokenization

• WSIs: Images are divided into patches and encoded using a pretrained image encoder (e.g., UNI or ResNet50) to generate pathology tokens.
• Genomics: Gene expression profiles are aggregated into pathway-level tokens (e.g., Hallmark pathways) to create compact, interpretable genomic factors.

2.2 The "Together" Stage: UOT-Guided Prototype Alignment

In this stage, both modalities are aligned to a shared set of prototypes ($K$ learnable vectors) to capture common survival-related structures.

• Unbalanced Optimal Transport (UOT): Instead of direct token-to-token matching, the model maps tokens from both modalities to the shared prototype bank using UOT. This allows uncertain or noisy tokens to remain diffuse or be routed to a "sink" rather than forcing a premature match.
• Curriculum Mass ($\rho$): To handle training instability, a curriculum strategy is employed where the total mass transported to real prototypes ($\rho$) is gradually ramped up from a small value to 1.0. This ensures that early in training, the model is tolerant of uncertainty, while later stages enforce decisive matching.
• Joint Assignment: Pathology and genomic tokens are concatenated and assigned to the shared prototypes via a single transport plan, allowing cross-modal evidence to interact under a unified geometry.
• Instance-Level Supervision: The UOT assignments serve as soft pseudo-labels to regularize token-to-prototype predictions via a cross-entropy loss ($\mathcal{L}_{instance}$).

2.3 The "Apart" Stage: Contrastive Diversification

After alignment, the model explicitly separates the modalities to preserve their unique characteristics and prevent representational collapse.

• Modality-Specific Anchors: Learnable anchors ($a^p$ for pathology, $a^g$ for genomics) are introduced to serve as modality identifiers.
• Anchor Refinement: A lightweight self-attention module refines the prototype tokens by appending the modality-specific anchor, ensuring the representations retain modality identity.
• Contrastive Regularization: An InfoNCE-style loss ($\mathcal{L}_{contrast}$) is applied to encourage features to stay close to their own modality anchor while remaining distant from the other modality's anchor. This prevents the modalities from collapsing into a homogeneous representation.

2.4 Prediction Head

The refined modality streams are combined using a co-attention module (transformer-style) that allows pathology to query genomics and vice versa at the prototype level. The resulting joint representation is mapped to a survival risk score using a linear head, trained with the Cox partial likelihood ($\mathcal{L}_{surv}$).

The total training objective is:
$$\mathcal{L}_{total} = \mathcal{L}_{surv} + \lambda_{contrast}\mathcal{L}_{contrast} + \lambda_{inst}\mathcal{L}_{instance}$$

3. Key Contributions

1. Identification of Over-Alignment Collapse: The paper highlights that aggressive cross-modal alignment can suppress modality-specific signals, leading to performance degradation compared to single-modality baselines.
2. The TTA Framework: A novel architecture that operationalizes the "Together-Then-Apart" principle. It balances cross-modal alignment (via shared prototypes and UOT) with modality-specific distinctiveness (via anchors and contrastive regularization).
3. Robust Alignment Mechanism: The use of unbalanced optimal transport with a curriculum mass schedule allows the model to handle noisy correspondences, intra-sample heterogeneity, and modality imbalance effectively.
4. Interpretability: The learned prototype structures reveal interpretable cross-modal patterns associated with clinical outcomes.

4. Experimental Results

The authors evaluated TTA on five TCGA cancer cohorts (BLCA, BRCA, STAD, CRC, KIRC) using paired WSI and genomic data.

• Performance: TTA achieved the highest overall Concordance Index (C-Index) of 0.693, outperforming the second-best multimodal method (MMP) by 2.6% and significantly outperforming unimodal baselines.
• Ablation Studies:
  • Removing the "Apart" stage caused a significant drop in performance on the CRC cohort (from 0.685 to 0.587), demonstrating that preserving WSI-specific signals is crucial when morphological cues are dominant.
  • Both the "Together" (alignment) and "Apart" (diversification) stages were shown to be complementary; enabling both yielded a 5.0% improvement over the baseline without these stages.
  • The curriculum mass schedule ($\rho$) and UOT soft assignments were critical for stability and performance compared to hard clustering (KMeans) or fixed-mass transport.
• Stratification: Kaplan-Meier analysis confirmed statistically significant separation between high-risk and low-risk groups across all five cohorts ($p < 0.05$).

5. Significance and Claims

The paper claims that effective multimodal survival models should not force heterogeneous signals into a single representation space indiscriminately. Instead, they should adopt a strategy that first aligns modalities to capture shared prognostic structures and then separates them to retain distinctive, modality-specific evidence.

The authors position TTA as a solution to the tension between cross-modal alignment and modality-specific information preservation. By demonstrating consistent improvements over recent state-of-the-art methods across diverse cancer types, the work suggests that balancing shared semantic alignment with modality-specific distinctiveness is a critical design principle for multimodal biomedical representation learning. The code is made available to facilitate further research in this direction.