Benchmarking Federated Learning in Edge Computing Environments: A Systematic Review and Performance Evaluation

This paper presents a systematic review and performance evaluation of Federated Learning in edge computing, benchmarking five leading algorithms across key metrics to identify trade-offs, highlight SCAFFOLD's superior accuracy and robustness versus FedAvg's efficiency, and propose a future research agenda to address challenges like data heterogeneity and energy limitations.

The Gist

Imagine you are the principal of a massive school with thousands of classrooms. Your goal is to teach the students a new skill, like solving complex math problems.

In the old way (Traditional Cloud Computing), you would ask every student to bring their homework, notebooks, and personal journals to your office. You would pile them all on your desk, study them, and then tell everyone the answers.

• The Problem: This is a privacy nightmare (students don't want to share their journals), it clogs your hallway (too much data to carry), and it takes forever.

Federated Learning (FL) is the new, smarter way. Instead of bringing the homework to you, you send the "answer key" (the model) to each classroom. The students solve the problems locally using their own notebooks. They only send you back the new things they learned (the updates), not their private journals. You combine these updates to make a better answer key and send it back out.

This paper is a systematic review (a giant report card) of how well different versions of this "classroom learning" system work when the classrooms are Edge Devices (like smartphones, smart fridges, or medical sensors) that are often far away, have weak internet, and run on batteries.

Here is a breakdown of the paper's findings using simple analogies:

1. The Four Main Ways to Organize the Class

The authors sorted all the different learning strategies into four buckets:

• Optimization (The Teaching Style): How do we make sure the students learn the same thing even if they have different textbooks?
  • FedAvg is the "Standard Teacher." It works great if everyone has similar books, but gets confused if the books are totally different.
  • SCAFFOLD is the "Tutor with a Compass." It uses special tools to stop students from drifting off-topic when their data is messy. The paper found this was the best at getting high scores.
• Communication (The Messenger): How do we send the updates without clogging the hallway?
  • Some methods "compress" the notes (like summarizing a long essay into a tweet) to save space.
  • Others let students send updates whenever they can, rather than waiting for a bell to ring (Asynchronous).
• Privacy (The Secret Keeper): Even if students don't share their journals, can a sneaky teacher guess what's inside just by looking at their notes?
  • Some methods add "static noise" (like a radio static) to the notes so no one can reverse-engineer the original journal. This is called Differential Privacy.
• Architecture (The School Layout):
  • Client-Server: One principal in the middle talking to everyone.
  • Peer-to-Peer: Students talking directly to their neighbors, with no principal. This is great if the principal's office burns down, but it can get chaotic.

2. The Report Card (Performance Evaluation)

The authors tested five top "teachers" (algorithms) using different "exams" (datasets like recognizing handwritten digits or Shakespeare's plays). Here is what they found:

• The All-Rounder (FedAvg): It's the most efficient. It uses the least battery and sends the smallest messages. It's like a bicycle: simple and fast, but it struggles on steep, rocky hills (messy data).
• The High-Achiever (SCAFFOLD): It gets the highest grades (accuracy) and handles rocky hills (messy data) the best. However, it's a bit heavier to carry (uses more energy and bandwidth).
• The Privacy Guard (FedAvg + DP): This version adds a "privacy shield." It's very safe, but the shield is heavy. The students learn slower and get slightly lower grades because the noise makes the math harder.

The Big Takeaway: There is no "perfect" teacher. You have to choose based on what your school needs.

• Need speed and battery life? Pick FedAvg.
• Need the highest accuracy on messy data? Pick SCAFFOLD.
• Need maximum privacy? Pick the Privacy Guard, but expect slower results.

3. The Current Hurdles (Why we aren't there yet)

Even though this technology is amazing, the paper points out three big problems that are still like "potholes" on the road:

1. The "Messy Classroom" Problem (Non-IID Data): In the real world, one student might only have pictures of cats, while another only has dogs. The "Standard Teacher" gets confused. We need smarter ways to teach mixed groups.
2. The "Weak Battery" Problem: Edge devices (like a smart watch) have tiny batteries. If the learning process drains the battery too fast, the device dies, and the learning stops. We need methods that are "energy-conscious."
3. The "Simulation vs. Reality" Gap: Most of these tests happen in a computer simulation (a video game version of the school). In the real world, the internet cuts out, phones run out of power, and people walk away. We need more tests done on real devices to see if the theory actually holds up.

Summary

This paper is a map and a compass for researchers. It tells us:

• Where we are: We have great tools (algorithms) to learn from data without invading privacy.
• What works best: SCAFFOLD is the smartest, but FedAvg is the most efficient.
• Where we need to go: We need to build better systems that can handle messy data, save battery life, and work reliably in the real world, not just in computer simulations.

In short, Federated Learning is the future of AI that respects your privacy, but we still need to tune the engine before it can drive the whole world.

Technical Summary

Here is a detailed technical summary of the paper "Benchmarking Federated Learning in Edge Computing Environments: A Systematic Review and Performance Evaluation" by Sales G. Aribe Jr. and Gil Nicholas T. Cagande.

1. Problem Statement

The rapid proliferation of IoT devices and edge computing has generated massive amounts of data at the network edge. While traditional cloud-centric architectures struggle with latency, bandwidth constraints, and data privacy regulations (e.g., GDPR), Federated Learning (FL) offers a decentralized alternative where models are trained locally on devices without sharing raw data.

However, integrating FL into real-world edge environments faces significant hurdles:

• Data Heterogeneity: Edge data is rarely Independent and Identically Distributed (non-IID), leading to "client drift" and model divergence.
• Resource Constraints: Edge devices suffer from limited battery life, computational power, and unstable network connectivity.
• Communication Bottlenecks: Frequent transmission of model updates creates high overhead in bandwidth-constrained networks.
• Lack of Standardized Benchmarking: Existing research often lacks a comprehensive, systematic comparison of FL algorithms across practical edge metrics (accuracy, energy, convergence, privacy).

2. Methodology

The authors conducted a Systematic Literature Review (SLR) following PRISMA and SALSA guidelines to synthesize and benchmark FL techniques for edge computing.

• Search Protocol: A comprehensive search was performed across major databases (IEEE Xplore, Scopus, etc.) for peer-reviewed papers published between January 2017 and June 2025.
• Selection Criteria: From an initial pool of 602 articles, 308 studies were selected based on inclusion criteria: empirical data, focus on edge computing, and reporting of quantitative performance metrics.
• Taxonomy Framework: The review categorizes FL techniques into four dimensions:
  1. Optimization Strategies: (e.g., FedAvg, FedProx, SCAFFOLD).
  2. Communication Efficiency: (e.g., quantization, compression, asynchronous methods).
  3. Privacy Mechanisms: (e.g., Differential Privacy, Secure Multiparty Computation).
  4. System Architectures: (e.g., Client-Server, Peer-to-Peer, Hierarchical).
• Benchmarking Approach: The authors evaluated five leading algorithms using standardized datasets (MNIST, CIFAR-10, FEMNIST, Shakespeare) and a multi-dimensional performance matrix.
• Metrics Evaluated:
  • Model Accuracy (%)
  • Convergence Time (Communication Rounds)
  • Communication Overhead (MB/round)
  • Energy Consumption (Joules/round)
  • Robustness to Non-IID Data
  • Privacy Mechanisms employed

3. Key Contributions

• Comprehensive Taxonomy: The paper provides a structured four-dimensional classification of FL techniques, offering a clear visual and conceptual map of the current landscape.
• Systematic Performance Matrix: It presents a comparative benchmark of five major FL algorithms (FedAvg, FedProx, SCAFFOLD, FedNova, and privacy-enhanced variants) across critical edge metrics.
• Identification of Trade-offs: The review explicitly highlights the inherent trade-offs between accuracy, privacy, energy efficiency, and communication costs.
• Gap Analysis: It identifies critical gaps in current research, particularly the lack of real-world testbeds, inconsistent energy reporting, and insufficient evaluation of fairness and privacy-utility trade-offs simultaneously.
• Future Research Agenda: The authors propose a roadmap for developing more robust, scalable, and reproducible FL systems for edge intelligence.

4. Key Results and Findings

The benchmarking results, synthesized from the reviewed literature, reveal distinct performance profiles for different algorithms:

• SCAFFOLD: Achieved the highest accuracy (84.7%) and demonstrated superior robustness to non-IID data. It utilized control variates to correct client drift, resulting in the fastest convergence (95 rounds). However, it incurred higher communication overhead (52 MB/round).
• FedAvg (Federated Averaging): Remains the most communication and energy-efficient baseline. While it has moderate accuracy (78.5% on CIFAR-10) and slower convergence (120 rounds), it is ideal for scenarios where bandwidth and battery life are the primary constraints.
• FedProx: Showed a strong balance between convergence speed and robustness under data heterogeneity, performing well on the FEMNIST dataset (81.2% accuracy).
• Privacy-Enhanced Variants (e.g., FedAvg + DP, SecureFed): Introducing privacy mechanisms like Differential Privacy (DP) or Secure Aggregation resulted in significant performance penalties. For instance, FedAvg + DP dropped to 74.1% accuracy and required 135 rounds to converge, highlighting the tension between privacy and model utility.
• Dataset Impact: The study noted that datasets with high non-IID severity (FEMNIST, Shakespeare) significantly challenge standard algorithms like FedAvg, whereas simpler datasets (MNIST) mask these issues.

Visual Insights:

• Radar Charts: Visualized that no single algorithm dominates all metrics. SCAFFOLD excels in accuracy/robustness, while FedAvg excels in efficiency.
• Word Clouds: Identified "non-IID," "privacy," and "energy efficiency" as the most dominant research themes.

5. Significance and Implications

• Guidance for Practitioners: The paper serves as a decision-making framework for system designers. It suggests that if accuracy and heterogeneity are the priorities, SCAFFOLD is preferred; if energy and bandwidth are critical, FedAvg is the optimal choice.
• Highlighting Real-World Limitations: The authors emphasize that current evaluations rely heavily on simulations. They argue for the urgent development of standardized real-world testbeds to capture the complexities of intermittent connectivity, device mobility, and hardware heterogeneity.
• Research Direction: The paper calls for future work to focus on:
  • Adaptive privacy mechanisms that reduce utility loss.
  • Energy-aware algorithms that dynamically adjust to device battery levels.
  • Unified benchmarks that simultaneously evaluate fairness, privacy, and efficiency.

In conclusion, this paper provides a foundational reference for the Federated Learning community, moving beyond theoretical proposals to a data-driven evaluation of how FL algorithms perform under the strict constraints of edge computing environments.