🤖 machine learning

FedTopo: Relation-Level Topology Sharing for Model-Heterogeneous Federated Learning

FedTopo addresses the challenge of non-aligned representation spaces in heterogeneous federated learning by encoding global knowledge as a relation-level class topology rather than absolute features, enabling reliable knowledge transfer and consistent performance improvements across diverse client architectures without inference overhead.

Original authors: Zhaoyang Ma, Zhihao Wu, Xin Gao, Lipo Wang, Youfang Lin, Jing Wang

Published 2026-07-30

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhaoyang Ma, Zhihao Wu, Xin Gao, Lipo Wang, Youfang Lin, Jing Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where everyone's smartphone, smartwatch, and home robot could learn together to get smarter, but without ever sharing their private photos or messages. This is the dream of Federated Learning. Instead of gathering all your data into one giant, vulnerable database, the learning happens right on your device. The devices just send tiny "updates" about what they learned to a central server, which mixes them together to create a smarter global brain.

However, there's a catch: not everyone has the same hardware. Some devices are powerful super-computers, while others are tiny, energy-saving chips. In the old days of machine learning, everyone had to use the exact same "brain architecture" to learn together. If your phone had a different brain than your neighbor's, they couldn't understand each other's updates. It's like trying to translate a book written in English into a language that uses completely different letters and grammar rules; the meaning gets lost in the translation. This paper tackles the messy reality of these mismatched devices trying to collaborate.

The Problem: Trying to Average Apples and Oranges

The authors of this paper, FedTopo, noticed that existing methods for helping these mismatched devices learn together were trying to force a square peg into a round hole. Most current methods try to share "prototypes." Imagine each device takes a snapshot of what a "cat" looks like in its own feature space and sends that snapshot to the server. The server then tries to average all these snapshots to find the "perfect cat."

But here's the rub: because every device has a different architecture (a different "brain"), the word "cat" might look like a fuzzy blob on one phone and a sharp geometric shape on another. When you try to average a fuzzy blob with a sharp shape, you don't get a better cat; you get a confusing mess that misleads everyone. The paper argues that trying to align these absolute coordinates (the exact position of a "cat" in a mathematical space) is a losing battle when the devices are so different.

The Solution: Sharing the Map, Not the Coordinates

Instead of trying to agree on where a "cat" sits in the universe, FedTopo suggests we just share how things relate to each other. Think of it like this:

Imagine you and a friend are both trying to navigate a city, but you have different maps. Your map says the library is "North of the park," while your friend's map says it's "Up the hill." You can't easily merge these maps because "North" and "Up" mean different things to each of you.

However, you can agree on the relationships. You can both agree that "The library is closer to the park than it is to the airport." This is a relationship, not a coordinate.

FedTopo works by having each device build a "relationship map" (a topology) of its classes. It doesn't say, "Class A is at coordinate (5, 10)." Instead, it says, "Class A is very similar to Class B, but very different from Class C." It calculates how similar a "cat" is to a "dog" versus a "truck" within its own local brain.

How It Works: The Trustworthy Messenger

The process is a bit like a group of detectives solving a mystery with different notebooks:

Local Detective Work: Each device looks at its own data and figures out how its classes relate. "Hey, my 'cats' and 'dogs' look kind of similar, but my 'cats' and 'trucks' are totally different." It sends this relationship map to the server.
The Trust Filter: The server knows that some devices might have very few pictures of "trucks," so their opinion on how "trucks" relate to "cats" might be shaky. FedTopo uses a special "reliability score" to weigh the messages. If a device has lots of data, its relationship map counts for more. If it has very little data, the server ignores its shaky guesses.
The Global Map: The server combines these weighted relationship maps into a single, super-reliable "Global Relationship Map."
The Lesson: This global map is sent back to the devices. Now, when a device is learning, it doesn't just look at the picture it's holding; it looks at the global map to see which other classes are "confusingly similar." It then practices harder on those specific confusing pairs to sharpen its skills.

What They Found

The authors tested this idea on three different datasets (CIFAR-10, CIFAR-100, and Tiny-ImageNet) using eight completely different types of neural network architectures (ranging from simple CNNs to complex ResNets). They simulated a chaotic world where data was unevenly distributed (some devices had only cats, others only trucks).

The results were clear: FedTopo consistently outperformed all the other methods.

On the CIFAR-10 dataset, it reached an accuracy of 87.38% in the standard uneven setting, beating the next best method by a solid margin.
In the even more difficult "pathological" setting (where devices had very few classes), it hit 86.26%, again leading the pack.
It worked just as well on the larger, harder datasets like CIFAR-100 and Tiny-ImageNet.

Crucially, the paper shows that this method is incredibly efficient. It doesn't require sending huge model files back and forth. It only sends tiny relationship tables (just 110 parameters to upload and 200 to download for a 10-class problem). Plus, it adds zero extra cost when the device is actually being used (inference), because the relationship map is only used during the training phase to help the device learn faster.

The Bottom Line

FedTopo proves that when devices are too different to agree on the exact "coordinates" of knowledge, they can still learn together by agreeing on the "relationships" between ideas. By focusing on how classes relate to one another rather than where they sit in a mathematical void, and by filtering out unreliable guesses, this method allows a diverse group of devices to build a smarter, more robust collective intelligence without needing to centralize their private data. It's a smarter way to learn together, even when everyone is speaking a slightly different language.

Technical Summary: FedTopo

Problem Statement

Federated Learning (FL) enables collaborative training across decentralized data silos without centralizing raw data. However, standard FL assumes identical client architectures, which is often impractical due to varying computational resources and deployment constraints. Model-heterogeneous FL addresses this by allowing clients to use different backbone architectures.

The core challenge in model-heterogeneous FL is transferring global knowledge across non-aligned representation spaces. Existing paradigms share knowledge via:

Partial Parameter Sharing: Requires aligned parameter spaces and compatible architectures.
Knowledge Distillation: Requires a shared output space or auxiliary public data to align predictions.
Prototype Sharing: Exchanges class-wise prototypes (feature coordinates).

All three paradigms encode knowledge in an absolute space (weights, logits, or feature coordinates) that must be aligned across clients. Under heterogeneous backbones, the same feature dimension may encode different semantic factors across clients. Consequently, directly averaging prototypes or parameters leads to unreliable global knowledge and misguides local training.

Methodology: FedTopo

The authors propose FedTopo (Federated Topology-guided Learning), a framework that shifts from sharing absolute coordinates to sharing relation-level class topology. Instead of asking "where does a class lie in feature space?", FedTopo asks "how do classes relate to one another within a client's local space?"

The framework operates in three stages:

1. Relation-Level Topology Construction (Client-Side)

Each client $k$ trains its local heterogeneous model on private data.

Prototype Extraction: The client computes class prototypes $p_{k,c}$ (mean feature representations) for classes present in its local dataset.
Relation Matrix Construction: Instead of uploading prototypes, the client computes a local relation matrix $R_k \in [0, 1]^{C \times C}$ . The entry $R_k(c, c')$ represents the cosine similarity between prototypes of class $c$ and $c'$ , rescaled to $[0, 1]$ .
Statistics: The client records class sample counts $n_k$ to quantify the statistical support for these relations.
Upload: Only the relation matrix $R_k$ and statistics $n_k$ are uploaded. The raw prototypes and feature coordinates remain local.

2. Reliability-Aware Global Topology Aggregation (Server-Side)

The server aggregates local relations into a global topology $R_g$ and a reliability matrix $S_g$ .

Masking: A binary mask $M_k$ excludes class pairs not observed on a specific client.
Reliability Weighting: A weight $w_k(c, c')$ is calculated based on the harmonic mean of the sample counts for the two classes. Relations supported by few samples are down-weighted.
Aggregation: The global relation $\tilde{R}_g$ is a reliability-weighted average of valid local relations.
Reliability Matrix: $S_g$ tracks the relative amount of evidence supporting each global relation pair.
Smoothing: An Exponential Moving Average (EMA) is applied to $R_g$ to reduce round-to-round fluctuations.
Broadcast: The server broadcasts $(R_g, S_g)$ to clients.

3. Topology-Guided Local Training (Client-Side)

Clients use the global topology to sharpen local decision boundaries without aligning feature spaces.

Guidance Selection: For a training sample with ground truth $y$ , the client selects the top- $q$ negative classes that are most topologically similar to $y$ (based on $R_g$ ) and most reliably supported (based on $S_g$ ).
Logit Modulation: The logits of these selected negative classes are increased by a factor $\gamma$ .
Loss Function: The local loss combines the standard cross-entropy loss with an auxiliary term that penalizes the model if it fails to distinguish the ground truth from these "hard" negative classes.
Inference: The global topology is used only during training. At inference time, clients use their original local models, incurring zero inference overhead.

Key Contributions

Relation-Level Knowledge Sharing: FedTopo is the first framework to share global knowledge as class relation topology rather than parameters, logits, or prototype coordinates. This relaxes the cross-client alignment assumption to an intra-client consistency assumption.
Reliability-Aware Aggregation: The method introduces a mechanism to down-weight unreliable relations caused by sparse data or weak support, ensuring the global topology is robust to non-IID data distributions.
Topology-Guided Training: A novel training objective that leverages global topology to emphasize confusing negative classes, improving local discrimination without requiring feature space alignment.
Efficiency: The framework achieves state-of-the-art performance with low communication costs (scaling with $O(C^2)$ rather than $O(C \times d)$ ) and no inference overhead.

Experimental Results

The authors evaluated FedTopo on CIFAR-10, CIFAR-100, and Tiny-ImageNet under Dirichlet and pathological non-IID settings, using eight heterogeneous backbones (ranging from CNN to ResNet-152).

Performance: FedTopo consistently outperformed baselines across all three knowledge-sharing paradigms (parameter, distillation, and prototype sharing).
- On CIFAR-10 (Dirichlet $\alpha=0.1$ ), FedTopo achieved 87.38% accuracy, surpassing the best baseline (FedSCE at 86.18%) by 1.2 points.
- On CIFAR-100 (Pathological), it achieved 59.30%, outperforming the best baseline (FedMRL at 57.45%) by 1.85 points.
Robustness: The performance gap widened as model heterogeneity and data skew increased, demonstrating that relation-level topology is more transferable than absolute coordinates.
Ablation Studies: Removing any component (reliability weighting, Top-q selection, missing-aware masking, or EMA) degraded performance, confirming the necessity of the full design.
Cost: FedTopo required only 110 parameters to upload and 200 to download per round on CIFAR-10, significantly lower than prototype-based methods (5,120 parameters). It added negligible training FLOPs and zero inference FLOPs.

Significance and Claims

The paper claims that FedTopo provides a more robust solution for model-heterogeneous FL by fundamentally changing the nature of shared knowledge. By shifting from absolute coordinate alignment (which fails under heterogeneous backbones) to relative relation consistency (which is preserved across different feature spaces), the method enables effective collaboration without requiring shared architectures, auxiliary data, or feature alignment.

The authors emphasize that this approach:

Avoids the "unreliable global prototypes" problem inherent in prototype sharing.
Operates effectively in data-free settings (no public auxiliary data required).
Delivers positive transfer even in highly personalized settings where local models are already strong.
Maintains low communication and computational overhead, making it practical for resource-constrained edge devices.

The work concludes that relation-level topology is a superior carrier for global knowledge in heterogeneous FL, offering a balance of transferability, robustness, and efficiency that existing paradigms lack. Future work is suggested for extending this to multi-modal and open-set scenarios.