🤖 machine learning

Towards Robust Federated Multimodal Graph Learning under Modality Heterogeneity

This paper proposes FedMPO, a robust federated multimodal graph learning framework that addresses modality heterogeneity and data isolation through topology-aware cross-modal generation, missing-aware expert routing, and reliability-aware aggregation to outperform existing methods in high-missing and non-IID settings.

Original authors: Sirui Zhang, Haonan Wang, Xunkai Li, Zekai Chen, Shumeng Li, Hongchao Qin, Rong-Hua Li, Guoren Wang

Published 2026-05-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sirui Zhang, Haonan Wang, Xunkai Li, Zekai Chen, Shumeng Li, Hongchao Qin, Rong-Hua Li, Guoren Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a group of friends trying to solve a giant, complex puzzle together. Each friend has a piece of the puzzle, but there are two big problems:

Privacy: They can't show their actual puzzle pieces to each other (because of privacy rules).
Missing Pieces: Some friends are missing entire sections of their puzzle. Maybe they lost the picture of the sky, or the part with the trees.

This is exactly the problem the paper FEDMPO tries to solve. It's about teaching computers (specifically "Federated Multimodal Graph Learning") to work together on these puzzles without sharing raw data, even when some data is missing.

Here is a simple breakdown of how they did it, using everyday analogies.

The Problem: The "Broken Puzzle" Dilemma

In the real world, data often comes in different forms (modalities), like text, photos, and numbers. Sometimes, a company or a person has the photo but not the text, or vice versa.

Old Method 1 (Centralized): Imagine everyone mailing their puzzle pieces to one big table to solve it. This works great, but in the real world, you can't mail your private photos or data to a central server.
Old Method 2 (Federated but Simple): Imagine everyone keeps their pieces at home and just sends a "summary" of their progress to a leader. The leader averages the summaries. But if one friend is missing half their puzzle, their summary is a guess. If the leader blindly averages these guesses, the final picture gets blurry and wrong.

The authors realized that existing methods either ignored the privacy rules or didn't know how to handle the "missing pieces" effectively when working in a group.

The Solution: FEDMPO (The Smart Team Leader)

The authors created a new system called FEDMPO. Think of it as a smart team leader who knows how to help friends fill in their missing puzzle pieces using clues from their neighbors, without ever seeing the actual missing pieces.

Here is how FEDMPO works in four steps:

1. The "Neighborhood Scout" (Topology-Aware Generation)

Usually, if you are missing a piece of a puzzle, you might guess what it looks like based on the pieces right next to it.

The Innovation: FEDMPO doesn't just look at the pieces right next to you. It looks at the whole neighborhood. If your friend is missing a "tree" picture, but their neighbor has a "forest" picture and the friend next to them has a "sky" picture, FEDMPO uses those surrounding clues to guess what the missing tree should look like.
Why it matters: It uses the structure of the network (who is connected to whom) to make better guesses than just looking at your own data.

2. The "Trust Filter" (Missing-Aware Expert Routing)

Now, imagine your friend makes a guess about the missing piece. Is that guess good? Maybe they have a lot of clues, so it's a great guess. Maybe they have very few clues, so it's a wild, bad guess.

The Innovation: FEDMPO has a "Trust Filter." It checks how confident the guess is.
- If the guess is highly confident, it mixes it with the real pieces.
- If the guess is low confidence (noisy), the filter says, "Hold on, this might be wrong," and gives it less weight.
- If the guess is terrible, the system ignores it and relies only on the structure of the puzzle itself (the connections) rather than the bad guess.
Why it matters: It prevents bad guesses from ruining the final picture.

3. The "Fair Vote" (Reliability-Aware Aggregation)

Finally, everyone sends their progress report to the leader.

The Problem: In old systems, the leader just counted how many pieces each person had. If a person had 1,000 pieces but 900 of them were bad guesses, their bad ideas would dominate the group's final picture.
The Innovation: FEDMPO's leader looks at the quality of the report, not just the size.
- "You have many pieces, but your 'Trust Filter' said they were shaky guesses. So, your vote counts less."
- "You have fewer pieces, but your guesses were very reliable. Your vote counts more."
Why it matters: It stops the group from being dragged down by unreliable data.

The Results: A Clearer Picture

The authors tested this system on six different "puzzles" (datasets) involving things like shopping items, dance videos, and social media posts. They simulated situations where data was missing and where everyone's data was very different from each other.

The Outcome: FEDMPO solved the puzzles better than any previous method.
The Gains: In the hardest scenarios (where data was very missing and very different), FEDMPO improved accuracy by up to 5.65%. That might sound small, but in the world of AI, that's a huge leap forward.

Summary

Think of FEDMPO as a smart team of detectives solving a mystery.

They can't share their private notebooks (Privacy).
Some detectives have torn pages (Missing Data).
Instead of guessing blindly, they use clues from their neighbors to fill in the gaps (Topology-Aware).
They check if a clue is reliable before using it (Trust Filter).
The team leader weighs the detectives' reports based on how reliable they are, not just how many words they wrote (Fair Vote).

The result is a solution that is much more robust, accurate, and trustworthy than previous methods, even when the data is messy and incomplete.

Technical Summary: Towards Robust Federated Multimodal Graph Learning under Modality Heterogeneity

1. Problem Formulation

The paper addresses the challenge of missing-modality robust learning in federated multimodal graphs (FMG). In real-world distributed systems, graph-structured data (e.g., social networks, recommendation systems) is often isolated across multiple clients due to privacy constraints, commercial competition, or platform ownership, making centralized training infeasible. Furthermore, these graphs frequently suffer from modality incompleteness, where nodes lack certain attributes (e.g., text, images, video) due to heterogeneous collection, storage limits, or user behavior.

Existing approaches fail to address this intersection effectively:

Centralized Multimodal Graph Learning: Methods that handle missing modalities assume centralized data access and ignore the challenges of cross-client knowledge sharing and generalization in federated settings.
Federated Multimodal Learning: While mature for non-graph data, these methods typically ignore graph topology and structural dependencies, treating samples as independent.
Federated Graph Learning: Most existing methods assume complete node features and do not explicitly model multi-granularity modality missingness.

The authors identify two critical bottlenecks in a naive generation-aggregation pipeline for this problem:

Topology-Isolated Local Completion: Client-side modality generation relies solely on local subgraphs, struggling to leverage global semantics. This leads to features inconsistent with cross-client structural semantics, potentially misleading graph message passing.
Reliability-Imbalanced Global Aggregation: Clients vary in modality availability and recovery reliability. Standard data-size-based aggregation (e.g., FedAvg) may bias the global model toward updates from clients with unreliable recovered signals or dominant but incomplete modality patterns.

2. Methodology: FEDMPO

The authors propose FEDMPO, a unified framework designed to improve local modality generation reliability and global robustness through a four-stage pipeline:

Stage 1: Feature and Graph Context Encoding

Mask Disentanglement: The framework distinguishes between natural missingness (no ground truth exists) and training-time masking (artificially hidden observed modalities). Only naturally observed modalities are artificially masked to provide reliable reconstruction targets.
Structural Anchors: To address the lack of global context, the model constructs modality-wise structural anchors for missing features by aggregating representations from neighboring nodes where the modality is visible. This provides a topology-derived prior for generation.
Context Encoding: Nodes are encoded using a graph encoder that incorporates either raw embeddings or structural anchors, ensuring the representation is enhanced by local graph context.

Stage 2: Topology-Aware Cross-Modal Generation

Context Bank: Instead of relying only on a node's own remaining modalities, the generator queries a context bank containing the node's other visible modalities and neighboring visible modality representations.
Attention Mechanism: A multi-head attention mechanism queries this bank to retrieve cross-modal and neighborhood evidence.
Warmup Strategy: A linear warmup factor ( $\gamma_t$ ) is employed. Early training rounds rely more heavily on stable structural anchors (low-variance topology priors) to reduce gradient instability, gradually shifting toward learned cross-modal generation as training progresses.
Objectives: The generator is supervised via a reconstruction loss on artificially masked modalities and a cross-modal alignment loss to ensure semantic consistency between observed and recovered modalities.

Stage 3: Missing-Aware Mixture-of-Experts (MoE) Fusion

Uncertainty Estimation: The model estimates a recovery uncertainty score for generated features based on the generated representation, cross-modal context, and structural anchors. Observed modalities are assigned zero uncertainty.
Expert Routing: A router dynamically weights two experts: an Observed Expert (reliable raw semantics) and a Recovered Expert (generated semantics). The routing decision considers visibility, uncertainty, and missing ratios at both node and client levels.
Reliability-Aware Fusion: Modalities are fused based on their reliability weights. Crucially, a structural fallback expert is introduced. If a node has a high missing ratio or high average uncertainty, the model adaptively shifts from multimodal fusion to a structure-only representation, preventing noisy recovered signals from propagating through the graph.

Stage 4: Task Prediction and Reliability-Aware Aggregation

Local Objective: Clients optimize a combined loss including task performance, reconstruction, alignment, and routing regularization.
Reliability-Aware Aggregation: Unlike standard FedAvg, the server aggregates updates using reliability-aware weights. Clients upload lightweight scalar statistics (average recovery uncertainty, reconstruction error, and client-level missing ratio) alongside model parameters.
Weighting Mechanism: The server computes a reliability score that exponentially down-weights clients with high uncertainty, high reconstruction error, or high missing ratios. This prevents unreliable client updates from dominating the global model, mitigating aggregation bias without excluding clients entirely.

3. Key Contributions

New Problem Definition: The paper formalizes the problem of missing-modality robust learning for federated multimodal graphs, highlighting the coupled challenges of graph structure, modality incompleteness, and federated heterogeneity.
Novel Framework (FEDMPO): It introduces a unified workflow integrating:
- Topology-Aware Cross-Modal Generation: Leveraging graph neighborhood structure to recover missing features.
- Missing-Aware Expert Routing: Filtering noisy recovered signals via uncertainty estimation and structural fallback.
- Reliability-Aware Aggregation: Down-weighting unreliable client updates based on recovery quality statistics.
Empirical Performance: Extensive experiments on 6 datasets across 3 tasks (Node Classification, Link Prediction, Modality Retrieval) demonstrate that FEDMPO outperforms strong baselines (including advanced FGL and multimodal FL methods).
- Achieves up to 4.10% performance gain in high-missing settings.
- Achieves up to 5.65% performance gain in highly non-IID settings.
- Demonstrates robustness to varying missing rates (30%–70%) and client heterogeneity.

4. Significance and Claims

The paper claims that FEDMPO is significant because it moves beyond treating modality recovery, graph reasoning, and federated optimization as separate components. Instead, it couples them into a single closed loop where:

Recovery Quality is explicitly modeled and used to guide fusion.
Graph Context is utilized to inform modality completion, addressing the "topology-isolated" limitation of existing methods.
Reliability Statistics are used to stabilize global aggregation, addressing the "reliability-imbalanced" limitation of standard federated learning.

The authors emphasize that while FEDMPO introduces moderate computational overhead (operating on compact hidden representations rather than raw data), it achieves a favorable trade-off between accuracy, robustness, and communication cost. The work suggests that explicitly modeling recovery reliability is essential for robust multimodal graph learning in federated environments, particularly when local updates are affected by heterogeneous modality availability.

The paper concludes by noting limitations, such as the need for more specialized modeling for complex real-world missingness patterns (e.g., missing-not-at-random) and the potential for future work in privacy-preserving aggregation of reliability statistics.