💻 computer science

Federated Cross-Modal Retrieval with Missing Modalities via Semantic Routing and Adapter Personalization

RCSR is a personalization-friendly federated learning framework that addresses non-IID data and missing modalities in cross-modal retrieval by utilizing prototype anchoring, a retrieval-centric semantic router, and lightweight client-specific adapters built on a frozen CLIP backbone.

Original authors: Hefeng Zhou, Xuan Liu, Sicheng Chen, Wutong Zhang, Wu Yan, Jiong Lou, Chentao Wu, Guangtao Xue, Wei Zhao, Jie Li

Published 2026-04-28

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Hefeng Zhou, Xuan Liu, Sicheng Chen, Wutong Zhang, Wu Yan, Jiong Lou, Chentao Wu, Guangtao Xue, Wei Zhao, Jie Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a massive, global digital library where people can search for a picture by typing a description, or find a description by uploading a photo.

To build this, you want to learn from thousands of different people (clients) around the world. However, there are three big problems:

The "Missing Piece" Problem: Some people only have photos; others only have text descriptions. They can't "match" them together to learn.
The "Different Tastes" Problem: One person might only have photos of cats, while another only has descriptions of cars. If you just average everyone's knowledge, the "cat" person and the "car" person might confuse each other.
The "Unfairness" Problem: Usually, the "big players" with tons of data dictate how the library works, leaving the "small players" with unique or limited data ignored.

The researchers created a system called RCSR to solve this. Here is how it works using three simple analogies:

1. The "Anchor" (Solving the Missing Piece)

Imagine you are teaching a child what a "dog" is. Usually, you show them a picture of a dog and say the word "dog." But what if the child only has the word "dog" and no pictures?

In RCSR, the system provides a "Global Anchor." It says, "I don't have a picture for you right now, but here is a 'ghost image' (a prototype) that represents the average idea of a dog that everyone else uses." This allows the person with only text to still learn how their words should align with the "ghost" images, keeping them in sync with the rest of the world.

2. The "Smart Traffic Controller" (Solving Different Tastes)

In standard systems, everyone’s opinion is treated as equally important. But in RCSR, there is a Semantic Router—think of it as a Smart Traffic Controller at a busy intersection.

Instead of just letting every car (every client's update) through, the controller looks at the "direction" the cars are heading. If a client sends an update that is totally nonsensical or contradicts the global goal (like saying a "cat" looks like a "car"), the controller says, "Whoa, you're heading the wrong way! I'm going to turn your volume down so you don't cause a pile-up." This ensures the global model stays stable and accurate.

3. The "Personal Assistant" (Solving Unfairness & Personalization)

To make sure no one is left behind, the system uses a Minimax Game. Think of this as a Fairness Referee. If the Referee notices that the "small players" are struggling to find anything in the library, they step in and say, "Hey, give more weight to these small players so the library works for them, too!"

Finally, for the users who still feel the global library doesn't quite fit their specific needs, the system gives them a Personal Assistant (the Adapter). This assistant doesn't change the whole library; they just sit next to you and tweak the results slightly to match your specific style. If the "Traffic Controller" already thinks you are a perfect match for the global library, your assistant stays quiet. But if you are a bit of an outlier, your assistant works harder to customize things just for you.

The Result

By using these three tools, the researchers built a system that is:

Efficient: It doesn't need to move massive amounts of data (it only shares small "notes").
Robust: It works even when half the people are missing one of the two modalities.
Fair: It works well for the "big" users and the "small" users alike.

Technical Summary: Federated Cross-Modal Retrieval with Missing Modalities via Semantic Routing and Adapter Personalization (RCSR)

1. Problem Statement

Federated Cross-Modal Retrieval (FCMR) aims to learn a shared embedding space for bidirectional image-text matching across decentralized clients while preserving privacy. However, the authors identify three critical challenges that existing federated learning (FL) methods fail to address simultaneously:

Missing Modalities: In real-world scenarios, clients often possess incomplete data (e.g., some clients have only images, others only text), which breaks standard contrastive learning (InfoNCE) that requires paired data.
Statistical Heterogeneity (Non-IID): Diverse semantic distributions across clients cause "alignment drift," where local updates conflict, distorting the global shared embedding space.
Fairness vs. Accuracy Trade-off: Standard aggregation (FedAvg) favors clients with large, high-quality datasets, often leading to poor retrieval performance for disadvantaged or single-modality clients.

2. Methodology

The proposed RCSR framework is built on a frozen CLIP backbone to ensure communication efficiency. It decomposes the solution into three core mechanisms:

A. Client-Side: Modality-Aware Training with Prototype Anchoring
To enable participation from single-modality clients, the authors introduce Prototype Anchoring.

Paired Clients: Use standard symmetric InfoNCE loss.
Single-Modality Clients: Since they cannot compute contrastive loss, they minimize an Anchoring Loss ( $L_{anchor}$ ), which pulls their local embeddings toward the global prototype of the missing modality (e.g., image-only clients are pulled toward the global text prototype).
Regularization: A prototype-proximal regularizer is used to prevent local models from drifting too far from the global semantic structure.

B. Server-Side: Retrieval-Centric Semantic Router
Instead of weighting clients by dataset size, RCSR uses a learnable Semantic Router ( $R_\phi$ ) that computes adaptive aggregation weights based on retrieval-specific statistics (local prototypes, update geometry, and loss).

Prototype Consistency Loss ( $L_{pc}$ ): Ensures the weighted combination of client prototypes approximates the global EMA (Exponential Moving Average) prototypes.
Alignment Consistency Loss ( $L_{ac}$ ): Ensures the "alignment direction" (the vector between image and text prototypes) of clients is consistent with the global alignment direction.

C. Robust-Fair Aggregation (Minimax Game)
To mitigate the impact of "bad" updates and ensure fairness, the framework employs a minimax optimization strategy. It treats the aggregation weights as a game where an adversary tries to maximize the loss of the worst-performing clients. This results in a fused weight $w_k$ that balances the router's quality signal ( $w_R$ ) with a fairness weight ( $q_k$ ), effectively protecting underperforming clients.

D. Optional Personalization (RCSR-p)
For clients whose data is highly unique or incomplete, the framework introduces Router-Guided Adapter Personalization. Each client maintains a lightweight local adapter. Crucially, the intensity of this adaptation is inversely proportional to the router's trust in the client: clients with low router weights (indicating high drift) receive stronger local adaptation to compensate for the global model's inadequacy.

3. Key Contributions

Novel Training Paradigm: Introduced prototype anchoring to allow unimodal clients to contribute to cross-modal alignment.
Retrieval-Centric Routing: Developed a transformer-based router that optimizes for the geometric consistency of the embedding space rather than just data volume.
Minimax Fairness: Integrated a robust-fair aggregation mechanism to stabilize training and improve worst-case client performance.
Communication-Efficient Personalization: Proposed a method to provide client-specific adaptation without increasing the communication overhead during the aggregation phase.

4. Experimental Results

The method was evaluated on Flickr30K, MS-COCO, IAPR TC-12, and MSR-VTT under severe conditions (50% missing modality rate, $\alpha=0.1$ non-IID).

Superior Accuracy: RCSR and its personalized version (RCSR-p) consistently outperformed state-of-the-art baselines (e.g., CreamFL, pFedMMA). On Flickr30K, RCSR-p achieved a significant gain over the strongest baseline.
Robustness: RCSR showed remarkable stability against increasing missing modality rates and extreme data heterogeneity.
Fairness: The minimax game successfully reduced the standard deviation of per-client performance, ensuring more equitable retrieval quality across the network.
Efficiency: The framework only requires training ~2% of the full CLIP parameters, making it highly practical for resource-constrained edge devices.

5. Significance

This work provides a comprehensive solution for the "fragmented data" problem in multimodal AI. By treating retrieval geometry as a first-class citizen in the federated optimization process, RCSR bridges the gap between theoretical federated learning and the practical realities of incomplete, heterogeneous, and privacy-sensitive multimodal data.