Federated Learning in the Wild: A Comparative Study for Cybersecurity under Non-IID and Unbalanced Settings

Technical Summary: Federated Learning in the Wild: A Comparative Study for Cybersecurity under Non-IID and Unbalanced Settings

Problem Statement
Machine Learning (ML) has demonstrated significant potential for network traffic analysis and intrusion detection. However, its effectiveness is often constrained by the need for representative, up-to-date datasets, which are difficult to aggregate due to privacy regulations and data-sharing restrictions. Federated Learning (FL) offers a solution by enabling collaborative model training while keeping sensitive data local. Despite this promise, the canonical FL algorithm, Federated Averaging (FedAvg), exhibits poor convergence and performance in heterogeneous environments characterized by non-independent and identically distributed (non-i.i.d.) data and unbalanced dataset sizes. These conditions are prevalent in cybersecurity, where different clients (e.g., network nodes) encounter distinct attack patterns and varying traffic volumes. Existing literature often lacks a comprehensive evaluation of alternative FL algorithms specifically within the context of network intrusion detection systems (NIDS) under such "pathological" settings.

Methodology
This study presents a systematic comparative evaluation of seven FL algorithms applied to network intrusion detection for Distributed Denial of Service (DDoS) attacks. The evaluation is conducted under a severe non-i.i.d. and unbalanced setting, described as a "pathological" scenario. In this setup:

• Data Distribution: 15 clients are utilized, where each client possesses a dataset containing benign traffic and exactly one specific type of DDoS attack.
• Imbalance: The volume of data across clients varies by more than four orders of magnitude. While individual client datasets are locally balanced between benign and malicious samples, the federation as a whole is highly unbalanced regarding both dataset size and attack-type representation.
• Algorithms Evaluated: The study compares the baseline FedAvg against six advanced methods:
  • General-purpose: FedProx (regularization), SCAFFOLD (control variates), and FedALA (adaptive local aggregation).
  • Cybersecurity-specific: DAFL (disparity-aware), FedSBS (greedy client selection), and FLAD (adaptive approach to DDoS).
• Experimental Setup: The experiments utilize recent benchmark datasets (CIC-IDS2017, CIC-IDS2018, and CIC-DDoS2019) covering 15 distinct DDoS attack types. The global model is a Multi-Layer Perceptron (MLP) trained using the Flower framework on CPU-only resources to reflect lightweight constraints.
• Metrics: Performance is assessed based on detection accuracy (F1 score), training duration, communication overhead, and client selection patterns. Ten independent runs with different random seeds ensure result reliability.

Key Contributions

1. Adaptation and Review: The authors review and adapt state-of-the-art FL algorithms, including those originally designed for computer vision, to the specific requirements of NIDS for DDoS detection.
2. Multi-dimensional Evaluation: Unlike studies focusing solely on final accuracy, this work provides a comprehensive comparison across detection performance, model generalization, training time, and communication overhead.
3. Analysis under Pathological Conditions: The study investigates algorithmic behavior under extreme heterogeneity where each client observes a unique attack profile, highlighting the strengths and limitations of various client selection, aggregation, and local training strategies.
4. Reproducibility: An open-source implementation of the evaluated algorithms and the experimental setup (including the dataset configuration) is provided to enable reproducibility.

Results
The comparative analysis yields several critical findings:

• Accuracy and Generalization: Most algorithms struggle to generalize effectively across all attack types, particularly for WebDDoS, LDAP, and LOIC attacks, which are underrepresented or have distinct feature distributions. FedAvg, FedProx, SCAFFOLD, and FedSBS generally achieve F1 scores below 0.8 for these specific attacks. FedALA performs poorly on TCP-based attacks. In contrast, FLAD demonstrates superior robustness, achieving high F1 scores (0.97 mean) across all attack types by dynamically adjusting training based on client performance.
• Training Time and Efficiency:
  • DAFL incurs the longest training duration because it requires all 15 clients to perform local training in every round, making the process bottlenecked by the slowest client.
  • FLAD significantly reduces total training time (despite high communication overhead) by adaptively selecting fewer clients per round and assigning more training steps only to those with lower accuracy, effectively excluding clients that have already converged.
  • Standard methods (FedAvg, FedProx, SCAFFOLD, FedALA) show comparable training times, with local operations accounting for approximately 60% of the total duration.
• Communication Overhead:
  • FedAvg, FedProx, FedALA, and FedSBS maintain low communication overhead similar to the baseline.
  • SCAFFOLD, DAFL, and FLAD introduce higher overhead. SCAFFOLD exchanges control variates; DAFL requires all clients to download the global model for validation; FLAD requires the transmission of accuracy scores and adaptive training parameters.
• Client Selection: Algorithms employing informed client selection (DAFL, FLAD) or adaptive workload distribution outperform those relying on random selection (FedAvg, FedProx, SCAFFOLD, FedALA) in terms of final model accuracy under non-i.i.d. conditions.

Significance and Claims
The paper claims that for cybersecurity scenarios characterized by highly heterogeneous and unbalanced data distributions, informed client selection and adaptive workload distribution are more effective mechanisms for improving detection performance than modifications to local training or global aggregation alone.

The authors argue that treating all clients equally (via fixed probability selection or uniform workload) fails to ensure that rare or difficult attack patterns contribute sufficiently to the global model. Instead, strategies that dynamically prioritize clients based on their current contribution to the learning process (as seen in FLAD and DAFL) allow the federation to utilize heterogeneous information more effectively. While these adaptive approaches incur additional communication and coordination costs, the study concludes that for network intrusion detection, where timely updates are critical to mitigating evolving threats, the trade-off favors adaptivity over strict communication efficiency. The findings suggest that future FL solutions for cybersecurity should prioritize mechanisms that address client-level heterogeneity directly through selection and workload management.