💻 computer science

FedHPro: Federated Hyper-Prototype Learning via Gradient Matching

FedHPro is a novel federated learning framework that introduces learnable hyper-prototypes optimized via gradient matching to preserve semantic consistency across clients, thereby achieving state-of-the-art performance by enhancing inter-class separability and intra-class uniformity in heterogeneous scenarios.

Original authors: Huan Wang, Jun Shen, Haoran Li, Zhenyu Yang, Jun Yan, Ousman Manjang, Yanlong Zhai, Di Wu, Guansong Pang

Published 2026-05-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Huan Wang, Jun Shen, Haoran Li, Zhenyu Yang, Jun Yan, Ousman Manjang, Yanlong Zhai, Di Wu, Guansong Pang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a group of students from different schools trying to solve a difficult puzzle together, but they can't share their actual puzzle pieces because of privacy rules. They can only send each other descriptions of what their pieces look like. This is Federated Learning: a way for computers to learn together without sharing private data.

However, there's a big problem. If Student A only has red pieces and Student B only has blue pieces, and they try to average their descriptions, the result is a muddy purple mess that doesn't really look like a red piece or a blue piece. In the paper's language, this is called data heterogeneity. The "global picture" they build becomes blurry and inaccurate because the students' local views are too different.

The Old Way: The "Average Description" Problem

Previous methods tried to fix this by having each student send a "prototype" (a summary description) of their pieces to a central teacher. The teacher would then average these descriptions to create a "Global Prototype."

The paper argues this is like trying to describe a "dog" by averaging a Chihuahua from one student and a Great Dane from another. The result is a weird, medium-sized dog that doesn't accurately represent either breed. This leads to semantic drift: the global signal gets confused and stops helping the students learn effectively.

The New Idea: "Hyper-Prototypes" (The Master Blueprint)

The authors, Huan Wang and colleagues, propose a new solution called FedHPro. Instead of just averaging descriptions, they introduce Hyper-Prototypes.

Think of a Hyper-Prototype not as a physical object, but as a Master Blueprint or a Mental Model that the teacher keeps in their head. This blueprint isn't made by averaging the students' muddy descriptions. Instead, it's built by looking at the direction the students are trying to learn.

Here is the creative analogy:

The Gradient: Imagine each student is pushing a heavy cart up a hill. The "gradient" is the direction and force of their push.
The Old Way: The teacher looks at where the students ended up (their final position) and averages those spots. If one student pushed north and another south, the average is "nowhere."
The New Way (Gradient Matching): The teacher looks at the force of their pushes. Even if the students are in different places, if they are all pushing "up the hill" (towards the correct answer), the teacher creates a Master Blueprint that aligns with that upward force.

The paper calls this Gradient Matching. The server (teacher) creates these Hyper-Prototypes by simulating the learning process of the real data without ever seeing the actual data. This ensures the "Master Blueprint" captures the true essence of the categories (like "cat" or "car") without being confused by the specific quirks of any single student's dataset.

How FedHPro Works: The Two-Step Dance

Once the server has these perfect Master Blueprints, it sends them back to the students. The students then use them in two special ways to improve their learning:

Hyper-Prototype Contrastive Learning (The "Push and Pull" Game):
Imagine the students are playing a game where they must stand close to their own team's "flag" (the correct Hyper-Prototype) and far away from other teams' flags.
- The Twist: The paper adds a "client-specific margin." This is like giving each student a personalized safety zone. It forces them to push their understanding of "cats" even further away from "dogs" than usual, making the decision lines between categories much sharper and clearer.
Hyper-Prototype Alignment Learning (The "Team Uniformity" Rule):
Even though the students are different, they need to agree on what a "cat" looks like. This module gently nudges every student's understanding to align with the Master Blueprint. It's like a coach saying, "Everyone, make sure your definition of a cat matches the official rulebook," ensuring that everyone is on the same page.

The Results: Why It Matters

The authors tested this on various difficult scenarios, such as:

Label Skew: Some students have mostly cats, others mostly dogs.
Quantity Skew: Some students have thousands of pictures, others have only a few.
Domain Skew: Some students have photos of cats taken in the dark, others in bright sunlight.

In these messy, real-world situations, FedHPro consistently outperformed the best existing methods.

Visual Proof: When they visualized the data, FedHPro's groups were tightly packed (students agreed on what a "cat" is) and widely separated from other groups (they clearly knew the difference between a cat and a dog). The old methods had messy, overlapping groups.
Plug-and-Play: The best part? They showed that you can swap the "old average descriptions" in other systems with their "Hyper-Prototypes" and instantly get better results.

Summary

FedHPro is a smarter way for AI systems to learn together without sharing private data. Instead of averaging confusing summaries, it builds a "Master Blueprint" by matching the learning direction of the data. This blueprint helps every student stay on the same page, leading to a global AI that is much more accurate and understands the world better, even when the data is messy and uneven.

Technical Summary: FedHPro: Federated Hyper-Prototype Learning via Gradient Matching

1. Problem Statement

Federated Learning (FL) enables collaborative training across distributed clients while preserving privacy but faces significant challenges due to data heterogeneity (non-IID data). In practical FL scenarios, clients possess data from diverse sources with varying distributions (label skew, quantity skew, and domain skew), leading to inconsistent local objectives and suboptimal global model convergence.

Existing prototype-based FL methods (e.g., FedProto) attempt to mitigate this by sharing global class prototypes to align local representations. However, these methods suffer from semantic drift:

Prototype-Level Bias: They update global prototypes by averaging local prototypes derived from biased client data. This aggregation inherits representation biases, causing the global signal to misalign with true class centers, particularly in "hard" domains or under extreme heterogeneity.
Insufficient Semantic Consistency: Simple averaging fails to capture the underlying semantic knowledge shared across diverse clients, resulting in ambiguous clustering and poor inter-class separability in challenging scenarios.

2. Methodology

The paper proposes FedHPro, a framework centered on Hyper-Prototypes and Gradient Matching.

2.1. Hyper-Prototypes via Gradient Matching

Instead of aggregating biased local prototypes, the authors introduce hyper-prototypes: a set of learnable, global class-wise prototypes ( $S_M$ ) optimized on the server.

Mechanism: The server aggregates gradients of prototypes computed from clients' real samples ( $g_k$ ), rather than the prototypes themselves.
Optimization: A set of learnable vectors simulates the hyper-prototypes. The server optimizes these vectors by minimizing a Gradient Matching (GM) loss ( $L_{GM}$ ), which maximizes the alignment between the gradients of the hyper-prototypes and the aggregated gradients of the real client samples.
Goal: This process simulates the optimization trajectory of centralized training (using all real samples) without accessing raw data, thereby distilling class-relevant semantic characteristics directly from real samples rather than prototype-level descriptors.

2.2. The FedHPro Framework

FedHPro integrates hyper-prototypes into local client training via two complementary modules:

Hyper-Prototype Contrastive Learning (HPCL):
- Constructs a mutual-contrastive learning objective using the hyper-prototypes.
- Introduces a client-specific margin ( $d_k$ ) calculated from pairwise distances between local prototypes to sharpen decision boundaries.
- Objective: Pulls sample embeddings closer to their assigned hyper-prototypes and pushes them away from other class hyper-prototypes, enhancing inter-class separability.
Hyper-Prototype Alignment Learning (HPAL):
- Applies a smooth regularization penalty at the feature level.
- Objective: Aligns sample embeddings with the averaged hyper-prototypes to enforce intra-class uniformity across clients, mitigating representation inconsistency caused by data heterogeneity.

The total local loss is a combination of standard Cross-Entropy, HPCL, and HPAL.

3. Key Contributions

Hyper-Prototypes: The introduction of learnable global prototypes optimized via gradient matching. Unlike traditional methods, these align with characteristics distilled from real samples, promoting semantic consistency across heterogeneous clients.
FedHPro Framework: A novel learning framework comprising HPCL and HPAL. HPCL enhances inter-class separability with client-specific margins, while HPAL ensures intra-class uniformity through feature-level alignment.
Theoretical and Empirical Validation:
- Provided a convergence analysis for FedHPro in non-convex settings.
- Demonstrated through extensive experiments that hyper-prototypes produce more semantically consistent global signals compared to centralized prototypes calculated via standard averaging.

4. Experimental Results

The authors evaluated FedHPro on diverse datasets (MNIST, CIFAR-10/100, TinyImageNet, HAM10000, Digits, Office-Caltech) under three heterogeneity scenarios: Label Skew, Quantity Skew, and Domain Skew.

Performance: FedHPro consistently outperformed state-of-the-art baselines (including FedAvg, FedProto, FedSA, FedTGP, and FedRCL).
- On the Digits dataset (Domain Skew), FedHPro achieved 84.80% accuracy, surpassing FedProto (80.41%) and FedSA (82.54%).
- On Office-Caltech, it achieved 64.52% (vs. 60.57% for FedSA).
- Under extreme long-tail distributions (Quantity Skew, $\rho=100$ ), FedHPro maintained superior performance where other methods degraded significantly.
Plug-and-Play Capability: Replacing global prototypes in existing methods (FedProto, FedTGP, etc.) with the proposed hyper-prototypes consistently improved their performance, validating the superiority of the representation.
Robustness: FedHPro demonstrated robustness under model heterogeneity (different feature extractors across clients), a scenario where prototype-based methods typically suffer catastrophic performance drops due to semantic drift.
Efficiency: The method incurs moderate communication and computation costs, comparable to other prototype-based approaches, by transmitting average gradients rather than full model parameters for prototype updates.

5. Significance and Claims

The paper claims that the core limitation of existing prototype-based FL is the reliance on skewed local representations to update global signals, leading to semantic drift. By shifting the optimization target to gradient matching with real samples, FedHPro effectively approximates the semantic consistency of centralized training without violating privacy.

The authors assert that FedHPro:

Solves the semantic drift issue inherent in averaging local prototypes.
Establishes a more generalizable decision boundary across diverse domains.
Provides a robust solution for heterogeneous FL scenarios, including extreme label imbalance and domain shifts.
Is applicable beyond image modalities, as demonstrated by preliminary results on text classification (AG News).

The work is positioned as an advancement in prototype-based FL that decouples semantic alignment from the biases of individual client data distributions.