FDRMFL: Multimodal Federated Feature Extraction Model Based on Information Maximization and Contrastive Learning

Technical Summary: FDRMFL – Multimodal Federated Feature Extraction Based on Information Maximization and Contrastive Learning

Problem Statement
The paper addresses the challenge of extracting predictive features from high-dimensional multimodal inputs (e.g., images, spectra, text) within a federated learning (FL) setting characterized by non-IID (non-identically distributed) data. In this scenario, raw data cannot leave local clients, samples are scarce, and distributions vary significantly across institutions. Existing approaches face critical limitations: unsupervised linear methods (PCA, TSVD, RP) ignore task-relevant labels and fail under non-IID conditions; Variational Autoencoders (VAEs) optimize reconstruction rather than prediction; and standard federated algorithms (FedAvg, FedProx, etc.) lack mechanisms for cross-modal alignment or task-driven feature retention in multimodal regression. Consequently, these methods suffer from representation drift, cross-modal misalignment, and the loss of predictive information.

Methodology: FDRMFL
The authors propose FDRMFL (Federated Dimensionality-Reducing Multimodal Feature extraction and Learning), a framework that jointly optimizes prediction accuracy and representation quality. The architecture consists of modality-specific encoders (e.g., CNNs, LSTMs, Transformers), a cross-attention fusion network, and a regression head. The core innovation lies in a unified local objective function comprising four terms:

1. MSE Prediction Loss ($\mathcal{L}_{pred}$): Minimizes the error between the fused representation and the continuous target variable.
2. Mutual Information Regularization ($\mathcal{R}_{mi}$): A correlation-based surrogate that maximizes the dependence between the fused representation and the target. Motivated by the Gaussian identity $I(X;Y) = -\frac{1}{2}\log(1-\rho^2)$, this term encourages the latent representation to retain task-relevant information by maximizing the absolute Pearson correlation ($|\hat{\rho}|$) between a projected representation and the target.
3. Symmetric KL Alignment ($\mathcal{R}_{kl}$): A penalty that aligns the latent distributions of different modalities before fusion. Assuming a homoscedastic Gaussian distribution for modality features, this term simplifies to minimizing the squared Euclidean distance between the means of different modality features, thereby closing the "modality gap."
4. Federated Contrastive Loss ($\mathcal{R}_{fcl}$): An InfoNCE-style loss that anchors local representations to the global consensus. It treats the representation from the previous global round as a positive sample and representations from earlier rounds (stored in a temporal history buffer) as negative samples. This constrains local updates to remain close to the global feature manifold, mitigating representation drift.

The global objective is a weighted sum of these four terms. The training follows a standard federated loop where clients perform local gradient descent on this composite loss, and the server aggregates parameters via sample-size weighting.

Key Contributions

• Unified Framework: FDRMFL is presented as the first unified framework to jointly address task-driven dimensionality reduction for continuous targets, cross-modal distributional alignment, and cross-round representation stability within a single federated regression setting.
• Correlation-Based MI Surrogate: The introduction of a low-variance, correlation-based mutual information surrogate specifically suited for small-sample federated regimes, avoiding the instability of high-variance neural MI estimators.
• Federated Contrastive Mechanism: A novel adaptation of contrastive learning to the fused multimodal space, utilizing a temporal history buffer to provide richer negative signals than previous single-modality approaches (e.g., MOON).
• Cross-Modal Alignment: A symmetric KL penalty designed to align latent distributions across modalities prior to fusion, addressing the specific challenge of non-IID multimodal data.

Experimental Results
The framework was evaluated on three synthetic datasets (with known ground-truth generative processes) and two real-world Near-Infrared (NIR) spectroscopy datasets (Tecator and Corn) under non-IID partitions.

• Performance vs. Traditional Baselines: FDRMFL reduced the mean Mean Squared Error (MSE) by 33.8% relative to the best traditional baseline (PCA) and by 43.0% relative to VAE in simulation. On real-world datasets, it consistently achieved the lowest MSE across all clients and targets.
• Performance vs. Federated Baselines: In comparisons with five established federated algorithms (FedAvg, FedProx, SCAFFOLD, MOON, FedBN), FDRMFL attained the lowest overall mean MSE across all tasks.
• Ablation Analysis: Removing any of the three regularization components ($\mathcal{R}_{mi}$, $\mathcal{R}_{kl}$, $\mathcal{R}_{fcl}$) resulted in increased error, with the removal of the contrastive loss ($\mathcal{R}_{fcl}$) causing the largest degradation (+7.8%), confirming the necessity of all components for stability and accuracy.
• Robustness: Sensitivity analysis demonstrated that performance remains stable across an order-of-magnitude variation in regularization coefficients, indicating the framework does not rely on fine-tuned hyperparameters.

Significance and Claims
The paper claims that FDRMFL effectively mitigates the "triple challenge" of non-IID data, high dimensionality, and multimodal misalignment in federated regression. By explicitly coupling the representation to the target (via MI), aligning modalities (via KL), and stabilizing updates (via contrastive learning), the framework achieves superior predictive accuracy and lower cross-client variance compared to existing methods. The authors position this work as a foundational step toward robust multimodal federated learning, particularly in domains like spectroscopy where data privacy and heterogeneity are paramount. The paper explicitly notes that while it provides baseline federated privacy (data locality), it does not yet incorporate formal differential privacy, identifying this as a direction for future work.