Technical Summary: ProMoE-FL

Problem Statement

The paper addresses the challenge of Multimodal Federated Learning (FL) with Missing Modalities in healthcare settings. In real-world clinical environments, data is fragmented across institutions due to privacy regulations and varying equipment availability. Consequently, participating clients often possess incomplete sets of modalities (e.g., some have only X-rays, others only text reports, and some have both).

Existing approaches to this problem face significant limitations:

• Centralized methods (e.g., prompt-based, generative, or dropout-based) are impractical for FL due to strict privacy constraints and communication efficiency requirements.
• Public Data Reliance: Methods like CAR-MFL require additional public multimodal datasets to compensate for missing client data, which is often unrealistic in healthcare due to privacy constraints and domain gaps.
• Naive Imputation: Approaches like zero-filling distort the joint feature space and bias models toward frequently observed modalities.
• Feature Synthesis Limitations: Methods like FeatImp synthesize features solely from observed modalities without modeling complementary information, while prototype-based methods like PmcmFL assign identical representations to all samples within a class, failing to capture intra-class variability crucial for medical data.

Methodology: ProMoE-FL

The authors propose ProMoE-FL (Prototype-conditioned Mixture-of-Experts for Federated Learning), a framework designed for robust missing-modality feature synthesis without relying on external public data.

1. Client-Aware Global Prototype Bank

To address client heterogeneity and distribution shifts, the framework constructs a global prototype bank that captures clinically meaningful modality priors across institutions.

• Prototype Construction: Each client maintains learnable, modality-specific prototypes ($P^k_m$) for each class. These are updated locally via a prototype alignment loss ($L_{proto}$) that aligns projected feature representations with class centroids.
• Global Aggregation: Unlike simple averaging, the server accumulates client prototypes to form a global bank ($P^{global}_m$). This bank serves as a "client-aware prior" in the latent space, guiding feature synthesis to account for distributional variability across clients.

2. Prototype-Conditioned Mixture of Experts (MoE)

For clients with missing modalities, a Prototype-Conditioned Decoder (PCD) synthesizes the missing features.

• Architecture: The PCD is implemented as a Transformer decoder. It takes the latent features of the available modality as the query and attends to the global prototype bank of the target (missing) modality to retrieve necessary context.
• Mixture of Experts: To handle complex, non-linear relationships between heterogeneous modality pairs, the system employs a Mixture of Experts ($\Phi_{MoE}$). This consists of multiple independent PCDs ($\phi_e$).
• Direction-Aware Routing: A Modality-Aware Router ($R$) computes a gating distribution conditioned on the instance context and modality indices (source $i$, target $j$). This dynamically routes the data to the optimal combination of experts, allowing the network to adaptively select the best synthesis strategy for specific missing-modality scenarios.

3. Federated Training Protocol

• Local Training: Clients optimize a combined objective of task loss ($L_{task}$) and prototype alignment loss ($L_{proto}$).
• Symmetric Reconstruction: Multimodal clients train the synthesis network using a symmetric reconstruction objective, learning to reconstruct missing features from available ones and vice versa.
• Communication: Clients transmit updated model parameters, projection heads, and local prototypes. The server aggregates these via FedAvg. The communication overhead is negligible (~0.3% of total parameters).

Key Contributions

1. Framework Proposal: Introduction of ProMoE-FL, a prototype-conditioned feature generation framework that enables missing modality synthesis in federated multimodal learning without external public data.
2. Novel Architecture: The integration of client-aware prototype attention to handle cross-client heterogeneity and a Mixture-of-Experts (MoE) architecture for direction-aware synthesis within a shared parameterization, preventing combinatorial parameter growth.
3. Robustness Validation: Extensive experiments demonstrating improved robustness in heterogeneous (non-IID) settings that mirror real-world clinical data silos, supported by both quantitative and qualitative analyses.

Experimental Results

The method was evaluated on four public chest X-ray datasets: MIMIC-CXR, NIH Open-I, PadChest, and CheXpert.

• Performance: ProMoE-FL consistently outperformed state-of-the-art baselines (Zero-filling, Uniform-filling, FeatImp, PmcmFL, and CAR-MFL) in both homogeneous and heterogeneous settings.
  • In homogeneous settings, it achieved the best performance in 6 out of 9 configurations.
  • In heterogeneous settings (representing realistic federated scenarios), it consistently outperformed all baselines across every configuration. Notably, it showed significant gains over PmcmFL, which suffered from degradation due to its reliance on single class-wise prototypes.
• Public Data Comparison: Even when compared against CAR-MFL (which utilizes public data), ProMoE-FL achieved competitive performance, demonstrating that its internal prototype mechanism effectively compensates for the lack of external data.
• Qualitative Analysis:
  • t-SNE/UMAP: ProMoE-FL demonstrated tighter integration between synthesized and ground-truth features with more precise class centroid alignment compared to FeatImp.
  • Rare Pathologies: The method maintained clear separation between class centroids for rare pathologies (e.g., Enlarged Cardiomediastinum, Fracture, Lung Lesion), whereas FeatImp exhibited mode collapse where different pathologies converged.
  • Expert Utilization: Gating distributions showed balanced expert utilization (centered near 0.5), indicating stable multi-expert collaboration rather than collapse to a single expert.
• Linguistic Heterogeneity: The study noted the inclusion of Spanish radiology reports (PadChest) alongside English datasets, marking a first in considering linguistic domain heterogeneity in this specific FL context.

Significance and Claims

The paper claims that ProMoE-FL provides a practical approach to addressing data heterogeneity in real-world clinical federated learning. By moving beyond simple feature imputation toward adaptive prototype-conditioned synthesis, the framework:

• Preserves diagnostic structures, including rare pathologies, ensuring synthesized data retains discriminative signals necessary for reliable diagnosis.
• Operates effectively without the unrealistic assumption of available public multimodal datasets.
• Successfully navigates the challenges of non-IID data and varying modality availability across medical centers, offering a robust solution for privacy-preserving multimodal healthcare AI.