Technical Summary: Med-MMFL

Problem Statement

Federated Learning (FL) offers a privacy-preserving mechanism for training models across decentralized medical institutions. However, the development of Multimodal Federated Learning (MMFL) in healthcare is hindered by a scarcity of standardized benchmarks. Existing efforts are fragmented: unimodal benchmarks (e.g., NIID-Bench, FLamby) lack multimodal capabilities, while existing multimodal benchmarks (e.g., FedMultimodal, FedVLM-Bench) often ignore medical data, limit modalities to two, or fail to combine realistic hospital-level silos with synthetic non-IID (non-Independently and Identically Distributed) settings. Furthermore, current evaluations lack coverage of diverse aggregation strategies, loss formulations, and regularization techniques required to understand how to scale beyond bimodal scenarios in complex clinical environments.

Methodology

The authors introduce Med-MMFL, a comprehensive benchmark designed to evaluate MMFL under realistic medical constraints. The framework integrates five publicly available medical datasets, encompassing 10 unique modalities (including text, pathology images, ECG, X-ray, radiology reports, and multiple MRI sequences) and spanning 2 to 4 modalities per dataset.

Data Partitioning and Tasks

The benchmark simulates three distinct federation scenarios:

1. Natural Partitioning: Utilizing actual institutional data splits where available (e.g., BraTS-GLI2024).
2. Synthetic IID: Uniform distribution of data across clients.
3. Synthetic Non-IID: Data heterogeneity simulated via Dirichlet distribution sampling to create skewed label distributions across clients.

The evaluation covers four distinct clinical tasks:

• Segmentation: Brain tumor segmentation (BraTS-GLI2024).
• Classification: Multi-label radiology finding classification (MIMIC-CXR-JPG).
• Modality Alignment (Retrieval): Zero-shot retrieval of X-rays based on ECG and lab data (Symile-MIMIC).
• Visual Question Answering (VQA): Pathology image reasoning (PathVQA) and EHR-based question answering (EHRXQA).

Algorithmic Adaptations

Med-MMFL evaluates six representative FL algorithms. Crucially, the authors generalize existing methods originally designed for unimodal or bimodal data to handle 2 to 4 modalities for fair comparison:

• FedAvg, FedProx, SCAFFOLD, FedNova: Standard aggregation and regularization techniques adapted for multimodal inputs.
• m-MOON (Multimodal MOON): A generalization of MOON that performs modality-wise contrastive alignment between local and global representations, acknowledging that different modalities may drift independently during local training.
• CreamMFL: An extension of CreamFL that utilizes pairwise inter- and intra-modal contrastive losses and global-local cross-modal aggregation to handle more than two modalities.

Key Contributions

1. First Comprehensive MMFL Benchmark: Med-MMFL is the first benchmark to jointly capture multimodal, medical, and federated aspects, covering 5 datasets, 10 modalities, 4 tasks, and 3 partitioning strategies.
2. Algorithm Generalization: The work extends state-of-the-art algorithms (specifically MOON and CreamFL) to support multimodal scenarios beyond the typical bimodal constraint, enabling unified evaluation.
3. Reproducibility: The authors release the complete benchmark implementation, including data processing and partitioning pipelines, to facilitate fair comparison and reproducibility in future research.

Experimental Results

Experiments were conducted on NVIDIA A100 GPUs across the defined datasets and partitioning strategies. Key findings include:

• No Universal Best Algorithm: No single algorithm consistently outperforms others across all configurations. Performance is highly dependent on the dataset, task, and data distribution.
• Impact of Heterogeneity:
  • FedProx demonstrated the most robust performance in non-IID settings, achieving the best results in 7 out of 16 settings, particularly excelling where data heterogeneity was high.
  • FedAvg and SCAFFOLD performed best primarily in IID settings.
  • FedNova showed exceptional performance on the BraTS-GLI2024 dataset (top results in 6 out of 7 configurations) but struggled on other datasets like Symile-MIMIC.
• Task Sensitivity:
  • Contrastive Learning Sensitivity: The Symile-MIMIC retrieval task showed a significant performance drop in non-IID settings across all algorithms. The authors attribute this to the reliance of contrastive learning on diverse batch samples, which are scarce in skewed non-IID partitions.
  • Robustness: VQA tasks (PathVQA, EHRXQA) demonstrated greater robustness to data heterogeneity, maintaining comparable performance across splits.
• Centralized vs. Federated: In most cases, the performance gap between centralized training and federated training was minimal (<1–1.5%). However, in specific non-IID scenarios, federated algorithms occasionally outperformed the centralized baseline, likely due to the implicit regularization provided by stochastic updates from diverse local distributions.

Significance and Claims

The paper positions Med-MMFL as a necessary step toward systematic understanding in medical MMFL. The authors claim that by providing a standardized platform with diverse modalities, tasks, and partitioning strategies, the benchmark addresses the fragmentation in current literature.

The significance lies in its ability to:

• Promote reproducibility and fair comparison of future MMFL methods under realistic medical settings.
• Reveal that algorithm performance is context-dependent, suggesting that a "one-size-fits-all" FL algorithm does not exist for multimodal healthcare.
• Highlight the specific challenges of applying contrastive learning and modality alignment in non-IID federated environments.

The authors explicitly state that the benchmark focuses on the federated aspects (aggregation of updates) rather than the design of fusion strategies, leaving the latter for future exploration. They also note that personalized FL and certain complementary SOTA methods are outside the current scope. The ultimate goal is to accelerate the development of robust, clinically relevant multimodal FL solutions.