🤖 machine learning

HERO: A Heterogeneity-Aware Benchmark Library for Federated Continual Learning

The paper introduces HERO, a heterogeneity-aware benchmark library for Federated Continual Learning that decouples task splits, client data distributions, and task sequences to enable reproducible evaluations revealing how method performance varies across different heterogeneity settings and can be masked by average metrics.

Original authors: Thinh T. H. Nguyen, Le-Tuan Nguyen, Minh-Duong Nguyen, Nhi Trinh, Anh Tran Nam Nguyet, Dung D. Le, Kok-Seng Wong

Published 2026-07-13

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Thinh T. H. Nguyen, Le-Tuan Nguyen, Minh-Duong Nguyen, Nhi Trinh, Anh Tran Nam Nguyet, Dung D. Le, Kok-Seng Wong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a massive class of students who are all in different rooms, using different textbooks, and learning at different speeds. This is the world of Federated Continual Learning (FCL). The goal is to train one smart "super-teacher" model that learns from all these students without ever seeing their private notebooks. But here's the catch: the students' lessons keep changing over time. One day they learn about cats, the next day about dogs, and the next about rockets. The teacher must remember the cats while learning the rockets, all while the students are scattered across the globe.

For a long time, scientists trying to test these "super-teachers" were playing a game of "compare apples to oranges." One researcher might test on a dataset of 100 pictures, another on 1,000; one might have all students learn in the same order, while another has them learning in chaos. It was impossible to tell if a method was actually smart or just lucky because the test was easier.

Enter HERO (Heterogeneity-Aware Benchmark Library for Federated Continual Learning). Think of HERO as a giant, perfectly calibrated obstacle course that every super-teacher must run through. Instead of letting researchers build their own messy tracks, HERO builds the track for them, ensuring everyone faces the exact same hurdles.

The Three Magic Knobs

The paper's main discovery is that previous tests were mixing up three different types of trouble into one big mess. HERO separates them into three distinct "knobs" you can turn to see exactly what breaks a model:

The "Who Gets What" Knob (Client Data Skew): Imagine a pizza party. If everyone gets an equal slice of every topping, that's easy. But what if one student only gets pepperoni, another only gets mushrooms, and a third gets a weird mix? This is client data skew. The paper uses a number called $\alpha$ to control this. A high $\alpha$ (like 100.00) means everyone gets a fair share. A low $\alpha$ (like 0.10) means the slices are wildly uneven.
The "Who Goes When" Knob (Task-Order Mismatch): Imagine a relay race. In a synchronized race, everyone runs leg 1, then leg 2, then leg 3 together. But in a chaotic race, Student A runs leg 1, then leg 3, then leg 2, while Student B runs leg 2, then leg 1, then leg 3. This is task-order mismatch. The paper uses a number called $\rho$ to control this. If $\rho$ is 0.00, everyone runs in perfect lockstep. If $\rho$ is 1.00, everyone is running in their own chaotic rhythm.
The "What's on the Menu" Knob (Task Split): This decides which topics (like "cats" or "dogs") make up each lesson.

By separating these, HERO lets researchers ask: "Does this model fail because the data is uneven, or because the students are confused about the order of lessons?"

The Big Surprise: Average Scores Lie

The paper ran a massive experiment using CIFAR-100 (a dataset of 100 image categories) and TinyImageNet (a smaller set of 100 image categories). They tested many different "super-teacher" methods.

Here is the big reveal: Average scores can hide a disaster.

In the "Easy" setting (where data is fair and everyone learns in order), one method called FedCBDR was the star. It had the highest average accuracy. But as soon as the researchers turned up the chaos (making data uneven and orders mixed), FedCBDR started to crumble.

Meanwhile, other methods like TagFed and LGA didn't look like the best in the easy setting, but they were the toughest survivors in the hard setting. They kept the "bottom 10%" of students (the ones with the hardest data and weirdest schedules) from failing completely.

The paper measured this using three specific scores:

Final Average Accuracy (AFA): The class average.
Average Forgetting (AF): How much the teacher forgot about old lessons.
Bottom-10% Client Accuracy (B10): How well the worst-off students did.

The results showed that a method could have a high AFA but a terrible B10. It's like a school where the top students get A+ grades, but the bottom 10% are failing the class. If you only look at the average, you'd think the school is great. HERO forces you to look at the bottom 10% to see the truth.

The Graph Experiment: Not Just Pictures

To prove HERO isn't just for pictures, the authors tried it on OGB-MolPCBA, a dataset about molecules (chemical structures). Instead of learning new types of molecules, the model had to learn from different groups of molecules that looked different (like different scaffolds).

They tested this by grouping molecules into 5, 15, or 25 different domains. As they made the groups smaller and more specific (increasing the number of domains), the models got worse. The "Average Precision" (AP) dropped, and "forgetting" increased. This proved that HERO's interface works for graphs and chemicals, not just images.

What the Paper Rules Out

The paper explicitly argues against the idea that you can judge these models with a single "Leaderboard" score. They show that a method that wins on a simple, synchronized test might lose badly when the real world gets messy. They also rule out the idea that "average performance" is enough; you must check the performance of the weakest clients to know if a method is truly robust.

How Sure Are We?

The authors are very sure about their findings because they didn't just guess; they measured it. They ran their experiments five times with different random seeds to make sure the results weren't just luck. They reported numbers like 53.91 ± 1.53 for accuracy on CIFAR-100 in the easy setting, showing exactly how much the results varied.

They found that in the "Joint-Hard" setting (where $\alpha = 0.10$ and $\rho = 1.00$ ), the best method (TagFed) achieved an AFA of 42.83 on CIFAR-100, while the easy-setting winner (FedCBDR) dropped to 39.74. These are measured facts, not just suggestions.

The Takeaway

HERO is like a new kind of gym for AI models. Instead of letting them run on a flat, empty track where they all look fast, HERO puts them on a track with hills, wind, and uneven footing. It shows us that the model that looks the fastest on a flat track might be the one who trips in the mud. By separating the "who gets what" from the "who goes when," HERO helps us build AI that is truly ready for the messy, unpredictable real world.

Technical Summary: HERO – A Heterogeneity-Aware Benchmark Library for Federated Continual Learning

1. Problem Statement

Federated Continual Learning (FCL) addresses the challenge of distributed clients learning from non-stationary data streams while retaining previously acquired knowledge. However, the field suffers from a significant "benchmark gap." Existing evaluations are difficult to compare because researchers simultaneously alter multiple variables: datasets, task splits, client data distributions, task sequences, model backbones, memory constraints, and reporting rules. Consequently, a method may appear superior not due to robustness, but because it was evaluated under a more favorable or less heterogeneous protocol. Current evaluations often fail to distinguish whether a method is robust to client data skew, task-sequence mismatch, or domain shifts, or if it only performs well under synchronized, easy settings.

2. Methodology: The HERO Framework

The authors introduce HERO, a heterogeneity-aware benchmark library designed to unify stream construction, method execution, and result reporting. The core design philosophy is the separation of three coupled choices that are typically conflated in prior work:

Task Split: Deciding which classes or domains form each task.
Client Data Split: Deciding the data distribution each client receives for each task.
Client Task Sequence: Deciding the order in which each client encounters tasks.

Stream Construction:
HERO generates benchmark streams $B(\xi, \alpha, \rho)$ where:

$\xi$ is a seed ensuring reproducibility.
$\alpha$ controls client data skew via a Dirichlet distribution ( $pt,c \sim \text{Dir}_K(\alpha \mathbf{1})$ ). Lower $\alpha$ values induce stronger skew (non-IID data).
$\rho$ controls task-order mismatch. It defines the deviation of individual client task sequences from a shared reference order $\pi^*$ . Higher $\rho$ values result in more asynchronous and mismatched task sequences.

The library includes three tiers:

HERO-Core: The primary comparable benchmark focusing on image-based Federated Class-Incremental Learning (FCIL) using CIFAR-100 and TinyImageNet.
HERO-Portability: Tests the interface's extension to other settings, specifically a Domain-Incremental Learning (Domain-IL) case study on OGB-MolPCBA (graph-based molecular data).
HERO-Extended: Stores additional dataset constructors and method implementations for future use.

Metrics:
HERO reports three primary metrics to capture different aspects of performance:

Final Average Accuracy (AFA): Measures overall final performance.
Average Forgetting (AF): Measures the drop in performance on old tasks.
Bottom-10% Client Accuracy (B10): Measures performance on the lowest-performing 10% of clients, highlighting robustness against heterogeneity.
For the graph-based portability study, Average Precision (AP) and AP-forgetting are used.

3. Key Contributions

The paper makes four specific contributions:

HERO Library: A unified, heterogeneity-aware benchmark library that separates stream construction from method execution, releasing fixed streams, configurations, and reporting scripts to ensure reproducibility.
Decoupled Stream Generator: A generator that isolates task split, client data split, and client task sequence, allowing researchers to control and inspect client data skew ( $\alpha$ ) and task-order mismatch ( $\rho$ ) independently.
HERO-Core Benchmark: A dense, image-based FCIL benchmark on CIFAR-100 and TinyImageNet featuring shared streams, controlled heterogeneity settings, and a representative pool of methods.
Portability Demonstration: A case study on OGB-MolPCBA showing that the HERO interface extends to graph-based Domain-IL, where scaffold-grouped molecular domains create distribution shifts without changing the prediction task.

4. Experimental Results

The authors evaluated representative FCL methods (e.g., FedAvg, GLFC, TagFed, LGA, FedCBDR) across an $\alpha$ - $\rho$ grid ranging from "Easy" (synchronized, balanced) to "Joint-Hard" (high skew, high mismatch).

Method Behavior Variance: Method performance is highly dependent on the heterogeneity setting. Methods that dominate in "Easy" settings (e.g., FedCBDR) do not necessarily remain the most stable under high heterogeneity. For instance, TagFed and LGA become more competitive as client data skew and task-order mismatch increase, particularly regarding B10.
Average vs. Bottom-Client Performance: Average accuracy (AFA) can mask weak performance on difficult clients. While FedCBDR often achieves high AFA, TagFed and LGA demonstrate superior Bottom-10% Client Accuracy (B10) in heterogeneous settings, indicating better protection for underperforming clients.
Distinct Failure Modes: The study reveals that client data skew and task-order mismatch cause different failure patterns.
- Client Data Skew ( $\alpha$ ): Primarily degrades B10, indicating that difficult clients are disproportionately affected by data imbalance.
- Task-Order Mismatch ( $\rho$ ): Primarily increases Average Forgetting (AF), suggesting that mismatched sequences create more temporal interference.
Portability: In the OGB-MolPCBA study, increasing scaffold-domain granularity (finer domains) reduced Average Precision and increased forgetting. Graph-based methods (POWER, MOTION) outperformed FedAvg, confirming that the HERO interface can describe distribution-shift streams beyond image-based FCIL.

5. Significance and Claims

The paper claims that HERO addresses the lack of controlled, comparable evaluation in FCL. By separating the sources of heterogeneity, HERO allows researchers to:

Determine if a method is robust to specific types of difficulty (skew vs. mismatch) rather than a generic "heterogeneous" setting.
Reveal that a single global leaderboard is insufficient; method rankings change significantly based on the specific heterogeneity configuration.
Expose failures on the "long tail" of clients (via B10) that average metrics hide.
Extend evaluation protocols to non-vision domains (like graphs) without mixing incompatible settings into a single score.

The authors emphasize that HERO is designed as a reusable library to support cumulative, reproducible, and setting-aware FCL evaluation, rather than a one-time experimental protocol. They explicitly state that while HERO documents method assumptions, it does not replace formal privacy analysis or deployment-level validation.