🤖 machine learning

A Comparative Study of Federated Learning Aggregation Strategies under Homogeneous and Heterogeneous Data Distributions

Original authors: Antonios Makris, Christos Dousis, Emmanouil Kritharakis, Stavros Bouras, Konstantinos Tserpes

Published 2026-05-13

📖 5 min read🧠 Deep dive

Original authors: Antonios Makris, Christos Dousis, Emmanouil Kritharakis, Stavros Bouras, Konstantinos Tserpes

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a group of friends trying to solve a giant puzzle together, but they can't share their actual puzzle pieces. Instead, they each work on their own section, write down a short summary of what they learned, and send just those notes to a central "Team Captain." The Captain then combines all the notes to create a master guide for the next round. This is Federated Learning: a way for computers to learn together without ever sharing their private data.

The big question this paper asks is: How should the Team Captain combine these notes?

If the Captain just takes a simple average of everyone's notes, does that work best? Or should they use a more complex method to handle the fact that some friends might be working on very different parts of the puzzle (some have mostly sky pieces, others mostly grass)?

Here is a breakdown of the study's findings using everyday analogies:

The Contestants: Different Ways to Combine Notes

The researchers tested several "strategies" the Captain could use to merge the friends' updates:

FedAvg (The Simple Averager): The Captain takes every note and calculates a straight average. It's fast and easy, like a quick vote.
FedAvgM (The Momentum Keeper): The Captain remembers what the group decided last time and uses that "momentum" to smooth out the new notes. It's like a runner who keeps their stride even when the terrain gets bumpy.
FedAdam & FedAdagrad (The Adaptive Learners): These captains are smart. They adjust how much they listen to each person based on how confident that person seems. If a note is confusing, they tweak the learning speed. They are like a coach who changes the training plan based on how each athlete is feeling that day.
FedMedian (The Filter): Instead of averaging, this Captain looks at all the notes and picks the "middle" value, ignoring the extreme outliers. If one friend sends a crazy note saying "the sky is green," the Captain ignores it because it's an outlier. This is great for filtering out noise or bad actors.
FedProx (The Restrainer): This Captain tells the friends, "Don't go too far from the last plan." It adds a rule to keep everyone's local learning close to the group's main goal, preventing anyone from drifting off too far.
DP (The Privacy Guard): This Captain adds a layer of "static noise" to the notes before combining them to ensure no one can guess the original puzzle pieces. It's like whispering the notes through a fan so the exact words are harder to hear.

The Test Conditions: Easy vs. Hard Puzzles

The researchers tested these captains in two scenarios:

Homogeneous (IID): Everyone has a similar mix of puzzle pieces (e.g., everyone has a bit of sky, grass, and trees). This is like a classroom where everyone studies the same textbook.
Heterogeneous (Non-IID): Everyone has very different pieces (e.g., one friend only has sky, another only has trees). This is like a classroom where one student only studied history and another only studied math.

They tested this on three "puzzles" (datasets):

MNIST & FMNIST: Simple puzzles (handwritten numbers).
CIFAR-10: A much harder, more complex puzzle (color photos of animals and objects).

What They Found (The Results)

1. There is no "One Size Fits All" Captain.
Just like you wouldn't use the same tool to fix a watch and build a house, no single strategy won every time.

On simple puzzles (MNIST): The Adaptive Learners (FedAdam) were the stars. They learned the fastest and got the highest scores, even when the data was messy.
On complex puzzles (CIFAR-10): Everyone struggled more, but the Simple Averager (FedAvg) and The Restrainer (FedProx) held up reasonably well. The Adaptive Learners actually got confused by the complexity and performed worse.
When data was messy (Non-IID): The Filter (FedMedian) was very stable. It didn't get thrown off by weird data, though it took a tiny bit longer to sort everything out.

2. The "Privacy Guard" (DP) has a heavy cost.
The strategy that added noise to protect privacy (DP) resulted in very poor scores. It's like trying to solve a puzzle while wearing thick foggy glasses; you can't see the pieces clearly enough to put them together. The paper notes that while it protects privacy, it hurts the model's ability to learn effectively.

3. Speed vs. Accuracy Trade-offs.

Speed: The simple strategies (FedAvg) were the fastest to combine notes. The complex ones (like the Filter or Privacy Guard) took a little longer to process, but the difference was small.
Accuracy: The "smart" strategies (Adaptive ones) were great for simple tasks but sometimes stumbled on complex ones. The "steady" strategies (FedProx) were reliable but didn't always reach the top score.

The Bottom Line

The paper concludes that you can't just pick one "best" way to combine learning updates.

If your data is simple and similar, use the Adaptive Learners.
If your data is messy or you have weird outliers, the Filter (Median) or Restrainer (Prox) is safer.
If you need to protect privacy, be prepared for a significant drop in performance.

The "best" choice depends entirely on how complex your puzzle is, how different your friends' data is, and how much you value speed versus accuracy. There is no magic bullet that works perfectly in every situation.

Technical Summary: A Comparative Study of Federated Learning Aggregation Strategies under Homogeneous and Heterogeneous Data Distributions

Problem Statement

Federated Learning (FL) has emerged as a paradigm for collaborative machine learning that preserves data privacy by keeping raw data localized on client devices. However, the performance of FL systems is critically dependent on the aggregation strategy used by the central server to combine local model updates. While traditional centralized learning suffers from privacy concerns and communication overhead, and isolated on-device learning lacks generalization, FL faces specific challenges regarding data heterogeneity (non-IID distributions).

The core problem addressed is the lack of a comprehensive understanding of how different aggregation mechanisms perform under varying degrees of data heterogeneity. Existing strategies range from simple averaging to robust estimators and adaptive optimizers, yet their trade-offs regarding learning accuracy, convergence stability, and system efficiency (computation and communication time) are not fully characterized across diverse datasets and distribution settings.

Methodology

The authors conducted a comprehensive experimental evaluation comparing seven widely used and state-of-the-art FL aggregation strategies:

FedAvg: Standard data-size-weighted averaging.
FedAvgM: FedAvg with server-side momentum.
FedAdam: Adaptive optimization using Adam at the server.
FedAdagrad: Adaptive optimization using Adagrad at the server.
FedMedian: Robust aggregation using the element-wise median.
FedProx: Aggregation with a proximal regularization term to constrain local drift.
Server-side Differential Privacy (DP): Adaptive clipping with noise injection for privacy.

Experimental Setup:

Datasets: MNIST, FMNIST, and CIFAR-10.
Distributions: Homogeneous (IID) and Heterogeneous (non-IID) data partitions generated using a Dirichlet distribution with concentration parameter $\alpha = 0.5$ .
Configuration: Simulations were run using the Flower framework with 10 clients over 25 communication rounds. A secondary large-scale experiment was conducted with 20 clients and 50 rounds on MNIST to verify consistency.
Models: Convolutional Neural Networks (CNNs) tailored for the specific datasets.
Optimizers: Client-side training utilized Adam (selected for superior convergence over SGD).
Metrics:
- Learning Performance: Centralized Accuracy and Loss.
- System Efficiency: Aggregation time, Training time, and Communication time per round.

Key Contributions and Results

1. Impact of Data Heterogeneity

The study confirms that data heterogeneity (non-IID) consistently degrades model performance across all strategies compared to IID settings. This degradation is most pronounced on the complex CIFAR-10 dataset. While all strategies suffer accuracy drops and increased loss in non-IID environments, the magnitude of the drop varies significantly by algorithm.

2. Strategy-Specific Performance

Adaptive Methods (FedAdam, FedAdagrad):
- FedAdam achieved the highest accuracy on MNIST under both IID and non-IID conditions, demonstrating the effectiveness of adaptive learning rates in stabilizing optimization under moderate heterogeneity. However, its performance deteriorated significantly on CIFAR-10, suggesting sensitivity to high dataset complexity.
- FedAdagrad followed a similar trend but generally exhibited lower accuracy on FMNIST and CIFAR-10 compared to other strategies, likely due to the learning-rate decay inherent to Adagrad limiting adaptation to complex tasks.
Robust and Regularization Methods (FedMedian, FedProx):
- FedMedian demonstrated the highest accuracy on FMNIST under both distributions. Its median-based aggregation effectively mitigated the influence of skewed client updates, providing stable performance. The trade-off was a slightly higher aggregation time due to sorting operations.
- FedProx provided stable and consistent performance across all datasets, acting as a "middle-ground" approach. The proximal term effectively constrained client drift, particularly benefiting performance on CIFAR-10.
Standard and Momentum Methods (FedAvg, FedAvgM):
- FedAvg served as a computationally efficient baseline, performing well on MNIST and FMNIST but struggling on CIFAR-10.
- FedAvgM showed high accuracy under IID conditions but experienced notable degradation under non-IID settings (especially on MNIST). The results suggest that server-side momentum can amplify biased gradient directions when client updates are skewed, leading to unstable convergence.
Privacy-Preserving (DP):
- The DP strategy (adaptive clipping with noise) exhibited substantially lower accuracy across all datasets and distributions. The noise introduced to ensure privacy significantly degraded the learning signal, preventing effective convergence. Additionally, it incurred the highest aggregation time due to clipping and noise injection operations.

3. System Efficiency

Aggregation Time: Generally remained low for most strategies. FedAvg and FedAvgM had the lowest overhead. FedMedian and DP incurred higher times due to sorting and noise injection, respectively, but these did not constitute a system bottleneck in the tested setups.
Training and Communication Time: These metrics were primarily driven by dataset complexity and model characteristics rather than the aggregation strategy itself. Differences between strategies were minimal.
Scalability: Increasing the number of clients (from 10 to 20) and rounds (25 to 50) confirmed the trends observed in the smaller setup. Adaptive methods continued to outperform others, and the performance gap between IID and non-IID settings narrowed slightly for adaptive methods, suggesting more communication rounds allow for better compensation of heterogeneity.

Significance and Claims

The paper concludes that no single aggregation strategy dominates across all scenarios. The effectiveness of a strategy is highly dependent on:

Dataset Characteristics: Complexity (e.g., CIFAR-10 vs. MNIST) significantly influences which algorithm performs best.
Degree of Heterogeneity: Strategies like FedAvgM may fail in highly skewed environments, while robust methods like FedMedian or regularization-based FedProx offer stability.
System and Privacy Requirements: There is a clear trade-off between privacy guarantees (DP) and model utility.

The authors emphasize that the choice of aggregation strategy should be guided by specific operational constraints and data properties rather than a "one-size-fits-all" approach. The study provides empirical evidence that while adaptive methods excel in moderate heterogeneity, robust estimators and regularization are necessary for stability in skewed distributions, and privacy mechanisms currently impose a significant cost on model utility.

Future work, as outlined by the authors, includes exploring more advanced robust mechanisms (e.g., Krum, Multi-Krum) and evaluating these strategies in more realistic FL deployment scenarios.