FedHyperM: Modality-Sensitive Federated Hypergraph Learning for Heterogeneous Multimodal Edge Intelligence

Technical Summary: FedHyperM

Problem Statement
Federated Learning (FL) enables edge devices to collaboratively train models without sharing raw data, addressing privacy concerns in Edge Intelligence (EC). However, practical edge deployments face significant modality heterogeneity: devices often possess different subsets of sensors (e.g., some have RGB cameras while others have only thermal or audio sensors) due to hardware differences, varying sensing conditions, or sensor failures.

In such heterogeneous settings, conventional FL aggregation strategies (like FedAvg) become unreliable. They assume that model parameters are learned from comparable input spaces. When devices train on different modality combinations, their learned parameters represent different evidence sources, making direct aggregation introduce noise and degrade global model performance. Furthermore, existing multimodal FL methods often treat missing modalities as data to be reconstructed or aligned, rather than designing frameworks that natively operate on varying available modality sets while preserving the high-order correlations between them.

Methodology: FedHyperM
The authors propose FedHyperM, a privacy-preserving federated multimodal learning framework designed specifically for edge devices under modality heterogeneity. The framework operates through three key processes:

1. Local Multimodal Feature Extraction & Freezing:
   Each edge device initializes a multimodal feature extractor using pre-trained models specific to its available modalities (e.g., BERT for text, CLIP for images, LSTM for signals, BEATS for audio). During the collaborative learning phase, these extractors are frozen. This ensures that the feature representation space remains consistent while the model focuses on learning the relationships between modalities.

2. Hypergraph-Based Multimodal Prediction (Edge Side):
   Instead of treating modalities as independent inputs, FedHyperM models them as a hypergraph where modality-specific features are nodes.

   • Adaptive Construction: The framework dynamically constructs hyperedges to capture high-order correlations. It defines "anchor nodes" (representing a specific modality) and dynamically connects them to "associated nodes" (other modalities) based on learned reconstruction coefficients.
   • Learning Objective: The system minimizes a reconstruction loss (measuring how well an anchor node can be reconstructed from its associated nodes) regularized by $L_1$ and $L_2$ norms to control sparsity and outlier impact.
   • Prediction: A multi-head attention mechanism aggregates hyperedge embeddings to update node embeddings, which are then passed through a Multi-Layer Perceptron (MLP) for prediction. This allows the model to learn complex, group-level relationships among the specific modalities available on that device.

3. Modality-Sensitive Aggregation (Server Side):
   To address the challenge of aggregating models trained on different modality sets, FedHyperM introduces a Modality-Sensitive Aggregation (MSA) mechanism.

   • Weighting Strategy: Instead of uniform averaging, the server calculates aggregation weights based on the Minimum Edit Distance (MinED) between the sets of available modality names of neighboring devices.
   • Logic: A smaller MinED implies higher similarity in modality availability, leading to a higher aggregation weight. This ensures that parameters from devices with similar modality configurations contribute more significantly to the global update, mitigating the noise caused by aggregating incompatible evidence.

Key Contributions

• Novel Framework: The proposal of FedHyperM, a collaborative framework specifically designed to facilitate federated multimodal learning across edge devices with heterogeneous modality availability.
• Hypergraph Learning Module: The design of a hypergraph-based prediction module that treats modality features as nodes and adaptively constructs hyperedges to capture high-order, non-pairwise correlations among multimodal features.
• MSA Mechanism: The development of a modality-sensitive aggregation strategy using Minimum Edit Distance (MinED) to weight model updates, ensuring that aggregation accounts for the specific modality composition of each edge device.
• Empirical Validation: Extensive experiments demonstrating that the framework effectively handles modality heterogeneity without requiring raw data sharing or complex imputation of missing modalities.

Experimental Results
The authors evaluated FedHyperM on the UR-FALL dataset (fall detection using RGB-depth video) under three different Modality Existence Rates (MER: $\rho = 0.5, 0.7, 0.8$), simulating varying degrees of modality missingness. They compared the framework against five baselines: FedAvg, FedCor, FedProx, AutoFed, and FedInMM.

• Performance Gains: FedHyperM consistently outperformed all baselines across all MER settings.
  • Accuracy: Improved average accuracy by 3.64% to 22.33% compared to the baselines.
  • F1-Score: Improved average F1-score by 2.91% to 26.82%.
  • Robustness: The framework showed particular strength in "Worst Accuracy" (W. Acc) and "Worst F1" (W. F1) metrics, indicating it stabilizes performance even for devices with the most limited modality sets.
• Ablation Studies:
  • Removing the hypergraph module (w/o HyperG) resulted in a performance drop of ~6.6% in average accuracy, confirming the necessity of capturing high-order correlations.
  • Replacing MSA with simple averaging (w/o MSA) caused a more significant drop (~9.15% in average accuracy), validating that modality-aware weighting is critical for heterogeneous aggregation.
• Hyperparameter Sensitivity: The model demonstrated stability, with optimal performance observed at $\alpha = 0.3$ and $\beta = 0.5$ (hyperparameters for $L_1$ and $L_2$ regularization).

Significance
The paper claims that FedHyperM addresses a fundamental gap in Edge Intelligence: the ability to collaboratively learn from distributed, heterogeneous sensor data without compromising privacy or requiring data imputation. By treating modality heterogeneity as a structural feature to be modeled via hypergraphs and weighted via edit distance, rather than a defect to be fixed, the framework enables more robust and accurate global models in real-world edge scenarios where sensor availability is inconsistent. The results suggest that explicitly modeling high-order modality correlations and respecting modality-specific aggregation weights are essential for the next generation of federated multimodal systems.