Technical Summary: RePercENT

Problem Statement

Multimodal machine learning aims to synthesize and reason over information from diverse sources (e.g., images, text, audio). While current state-of-the-art approaches focus on cross-modal alignment and fusion, these methods often result in entangled representations where shared and modality-specific information are mixed. This limits interpretability and the ability to identify which modalities contribute unique versus redundant information.

A critical bottleneck in existing disentangled representation learning is scalability. Most prior work is confined to the two-modality regime. As the number of modalities ($M$) increases, the number of possible shared and unique factors grows exponentially ($O(2^M)$), making them difficult to model. Furthermore, tractable optimization objectives that explicitly characterize all cross-modal interactions beyond two modalities are limited. Real-world data often presents a trade-off between minimizing redundancy and preserving relevant information, and capturing only the Minimal Necessary Information (MNI) is crucial to avoid representational collapse.

Methodology: RePercENT

The authors propose RePercENT (Reduced-complexity Perceiver-based disENTanglement), a self-supervised framework designed to scale disentangled representation learning beyond two modalities.

Core Architecture

RePercENT operates directly on pre-extracted embeddings from foundation models (FMs), eliminating the need for extensive joint pre-training and making the approach modality-agnostic.

1. Pairwise Decomposition: Instead of modeling all $2^M$ atomic factors, RePercENT models pairwise interactions. For every modality pair $(i, j)$, it derives:
   • Unique components ($u_{ij}$): Information unique to modality $i$ relative to $j$.
   • Shared components ($s_{ij}$): Information shared between $i$ and $j$.
     This reduces component complexity from $O(2^M)$ to $O(M^2)$, while preserving all cross-modal interactions.
2. Disentanglement Module ($D_i$): Each modality's composite representation $Z_i$ is processed by a dedicated module. This module utilizes a learnable latent array $H_i$ with $N = 2(M-1)$ slots.
3. Routing Mechanisms:
   • Semantic Encoding: Each latent slot is augmented with learnable pair encodings (identifying the modality pair) and type encodings (identifying "unique" or "shared").
   • Group Slot Attention: A structural routing mechanism where slots corresponding to a specific pair $(u_{ij}, s_{ij})$ compete over the same input embeddings. This enforces specialization within the pair while maintaining independence between different pairs.
4. Scalability: The framework requires only one encoder per modality, achieving linear scaling $O(M)$ in terms of structural complexity, compared to the $O(M^2)$ complexity of methods using separate encoders for each representation.

Optimization Objective

The framework employs a joint optimization objective to simultaneously derive shared and unique components, rather than a sequential decomposition. For each pair $(i, j)$, the objective maximizes:
$$J(\theta_{ij}) = \alpha(L_{si} + L_{sj}) + L_{ui} + L_{uj}$$
Where:

• Shared Loss ($L_{si}, L_{sj}$): Encourages shared representations to capture cross-modal relevance ($I(s_{ij}; X_j)$) while minimizing information explainable only by the source modality ($I(s_{ij}; X_i | X_j)$).
• Unique Loss ($L_{ui}, L_{uj}$): Preserves source-modality information ($I(u_{ij}, s_{ji}; X_i)$) while penalizing overlap between unique and shared components across modalities ($I(u_{ij}; s_{ji})$).

The information-theoretic terms are approximated using tractable surrogate losses: InfoNCE for mutual information, KL-divergence for conditional mutual information, and a cross-covariance penalty for redundancy.

Theoretical Guarantees

The paper provides formal optimality guarantees:

1. Exact Optimality: Under the condition that the Minimal Necessary Information (MNI) is attainable and the weighting parameter $\alpha > \lambda$, the joint objective admits the same global optimum as the ideal step-by-step optimizer.
2. Near-Optimality: When MNI is unattainable (a common real-world scenario), the joint objective is proven to be near-optimal, with the solution bounded by a tractable constant related to the information gap.

Experimental Results

The authors evaluated RePercENT across three domains: synthetic data, figurative language understanding (IRFL), and multimodal oncology (TCGA).

1. Scalability and Component Recovery (Synthetic):

   • RePercENT demonstrated linear scaling in parameter count as modalities increased ($M=2$ to $5$), whereas baselines (MLP, GRU, gMLP) required exponential increases or suffered from component collapse.
   • RePercENT maintained competitive disentanglement performance (high intra-component prediction, low cross-component leakage) across all modality counts, while baselines like GRU and MLP failed to recover unique components as $M$ grew.
   • It achieved comparable performance to gMLP with significantly fewer parameters (approx. 47% lower at $M=3$) and FLOPs.

2. Figurative Language (IRFL):

   • On the task of matching figurative text (idioms, metaphors, similes) to images, RePercENT outperformed fine-tuned CLIP variants and other disentanglement baselines.
   • The method showed particular strength in handling abstract, context-dependent tasks (idioms and metaphors) where isolating shared components proved beneficial.
   • Ablation studies confirmed that both semantic encoding and group slot attention are critical for performance.

3. Multimodal Oncology (TCGA):

   • Using embeddings from Clinical, Pathology, WSI, and Molecular data, RePercENT improved cancer-type classification accuracy over the original HONeYBEE embeddings for most cancer types.
   • Notably, for cancer types where raw molecular embeddings were uninformative (e.g., KIRC, LGG), the decomposed shared representations recovered significant predictive structure.
   • Robustness to Missing Modalities: In experiments where WSI data was progressively dropped, RePercENT maintained high Macro-F1 scores, whereas fusion baselines (including those trained with dropout) degraded sharply. This is attributed to the inference structure where components are derived solely from the available modality.

Significance and Claims

The paper claims that RePercENT provides a principled framework that bridges information-theoretic foundations with practical, scalable implementation. Its primary contributions are:

• Scalability: It successfully overcomes the exponential scaling bottleneck of multimodal disentanglement, enabling the extraction of unique and shared factors for more than two modalities with linear complexity.
• Plug-and-Play Design: By operating on pre-extracted embeddings, it is agnostic to modality types and backbone architectures, facilitating easy integration into existing pipelines.
• Theoretical Grounding: It offers formal guarantees of optimality (both exact and near-optimal) for its joint optimization approach, ensuring theoretical integrity alongside empirical success.
• Robustness: The architecture inherently handles missing modalities without complex imputation, making it suitable for real-world clinical and scientific settings where data completeness is not guaranteed.

The authors conclude that RePercENT paves the way for more reliable, interpretable, and mechanistically grounded multimodal representations, moving beyond simple alignment to a granular understanding of how different modalities interact.