💻 computer science

A Statistical Comparison of SCQ-SSA, PSO and ARO for Optimizing Bi-LSTM-Based Network Intrusion Detection

This study establishes a reproducible protocol for comparing metaheuristic optimizers (SCQ-SSA, PSO, and ARO) in tuning a Bi-LSTM-DRN for network intrusion detection, revealing that while SCQ-SSA achieved slightly higher metrics, the differences were not statistically significant and the model ultimately underperformed a Random Forest baseline, highlighting the value of the evaluation framework over the specific classification results.

Original authors: Mohd. Asgar Ali

Published 2026-10-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mohd. Asgar Ali

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast, invisible architecture of the internet, data flows like water through pipes, carrying everything from harmless emails to malicious attacks designed to shut down systems. To protect these digital networks, security experts use automated systems called intrusion detectors, which act as vigilant sentinels, constantly scanning traffic for signs of trouble. For years, these systems relied on straightforward rules or basic statistical patterns to spot intruders. However, as attacks have become more sophisticated, researchers have turned to a more powerful approach: teaching computers to learn from the data itself. This involves using deep learning, a type of artificial intelligence that mimics the way the human brain processes information, to recognize complex patterns that traditional methods might miss. The challenge, however, is that these smart systems have many internal settings, or "knobs," that need to be turned to just the right position to work effectively. Finding the perfect combination of these settings is difficult, and researchers often use mathematical strategies called optimization algorithms to search for the best configuration, much like a hiker trying to find the highest peak in a foggy mountain range.

A recent study by Mohd. Asgar Ali and his colleagues at Lovely Professional University and other institutions in India set out to test three different search strategies to see which one was best at tuning these deep learning systems for network security. The researchers focused on a specific type of advanced AI model known as a Bidirectional Long Short-Term Memory network, which is designed to look at data from multiple angles to understand context better. They wanted to see if a new, complex search method called the Chaos-Sine-Cosine-Quantum Salp Swarm Algorithm could outperform two established rivals: Particle Swarm Optimization, which mimics the flocking behavior of birds, and Artificial Rabbits Optimization, which is inspired by how rabbits hide and forage. To ensure a fair test, the team used a massive, real-world dataset of network traffic containing nearly three million records of both normal activity and various types of cyberattacks. They did not just run the tests once; they repeated the entire experiment twenty times for each method to ensure the results were reliable and not just a lucky fluke.

The researchers discovered that while the new algorithm did perform slightly better on average, the difference was so small that it was not statistically significant. In fact, the new method achieved an accuracy of about 85 percent, while the older methods came very close behind. More surprisingly, when the team compared these complex, deep learning models against a much simpler, older technique called Random Forest, the simple method won handily, achieving an accuracy of over 98 percent. This finding suggests that for this specific type of network security task, the elaborate deep learning architecture and the fancy new search algorithm did not provide the expected advantage. The study also revealed that the deep learning models struggled to generalize their knowledge; when tested on traffic from a different day than the one they were trained on, their accuracy dropped significantly, indicating they had memorized the specific patterns of the training data rather than learning the underlying rules of the attacks.

Ultimately, the most important contribution of this work was not the discovery of a new, superior security tool, but the rigorous and transparent way the researchers compared the different methods. By strictly controlling every variable—from the size of the search group to the specific data used for testing—they created a reproducible protocol that future researchers can use to fairly evaluate new ideas. The study concluded that the main strength of their deep learning model came from its structural design, specifically how it processed information in both directions and how it connected different layers, rather than from the specific weighting of the data features. While the new algorithm showed promise, the results indicate that in the world of network intrusion detection, simpler, well-established methods can still outperform highly complex, cutting-edge systems, and that the real value lies in having a clear, fair standard for measuring progress.

Technical Summary: A Statistical Comparison of SCQ-SSA, PSO, and ARO for Optimizing Bi-LSTM-Based Network Intrusion Detection

1. Problem Statement

The study addresses the challenge of optimizing hyperparameters and feature weights in deep learning-based Network Intrusion Detection Systems (NIDS) for cloud-oriented environments. While deep sequence models like Bidirectional Long Short-Term Memory (Bi-LSTM) networks are increasingly used to model dependencies in network flow features, their performance is highly sensitive to the choice of meta-heuristic optimizer and feature engineering strategies.

The specific gap identified is the lack of rigorous, statistically controlled benchmarks comparing novel optimizers against established baselines within an identical pipeline. Previous studies often introduce new optimizers (such as the Chaos-Sine-Cosine-Quantum Salp Swarm Algorithm, or SCQ-SSA) without direct, repeated-run comparisons against standard algorithms like Particle Swarm Optimization (PSO) and Artificial Rabbits Optimization (ARO) on genuine network traffic datasets. Furthermore, the contribution of specific architectural components (e.g., bidirectionality, residual connections) and feature weighting mechanisms (OWNSI) often remains unisolated in ablation studies.

2. Methodology

The research proposes a reproducible protocol to evaluate three optimizers—SCQ-SSA, PSO, and ARO—on a unified NIDS architecture.

Dataset: The experiments utilize the genuine CIC-IDS2017 dataset, comprising 2,830,743 labelled flow records across 14 attack categories. Due to computational constraints (single-CPU environment), a stratified working sample of 9,000 flows was created. This sample intentionally over-represents rare attack classes (e.g., Heartbleed, SQL Injection) to ensure their observability during training, while maintaining the overall benign-to-attack ratio of the original population.
Feature Extraction & OWNSI: Five key features (Packet Size, Connection Duration, Source Bytes, Destination Bytes, and a Protocol proxy via Destination Port) are extracted. These are processed through an Optimized Weighted Network Security Index (OWNSI) module. The OWNSI assigns a fixed weight vector ( $w \in [0.1, 1.0]$ ) to these features, determined offline by the optimizer, rather than adapting dynamically during inference.
Architecture (Bi-LSTM-DRN): The core classifier is a Bidirectional LSTM Deep Residual Network. The five weighted features are treated as a sequence of length 5. The author explicitly notes that this sequence does not represent temporal evolution over time (as the features are dimensions of a single flow) but is used to allow the Bi-LSTM to learn inter-feature dependencies. The architecture includes a residual (skip) connection to mitigate vanishing gradients.
Optimization Protocol:
- Search Space: A 7-dimensional space consisting of 5 OWNSI feature weights, the Adam learning rate, and the dropout rate.
- Budget: All optimizers use a population size of 10 and 10 iterations. SCQ-SSA performs 120 fitness evaluations per run, while PSO and ARO perform 110.
- Repetition: To ensure statistical robustness, each optimizer is executed 20 times with different random seeds.
- Fitness Function: A multi-metric function equally weighting Accuracy, Precision, Recall, and F1-score (0.25 each).
- Evaluation: A fixed, stratified test set of 1,500 flows is generated once and reused for all experiments, including ablation studies and baseline comparisons.

3. Key Contributions

Controlled Benchmarking: The study provides a direct, head-to-head comparison of SCQ-SSA against PSO and ARO under identical search spaces, fitness functions, and fixed test sets, moving beyond isolated performance claims.
Statistical Rigor: Unlike many prior works, this study employs 20 independent repetitions and reports comprehensive statistical metrics, including Friedman tests, paired t-tests, Wilcoxon signed-rank tests, Cohen's d effect sizes, and confidence intervals.
Ablation Analysis: A six-configuration ablation study isolates the specific contributions of bidirectionality, residual connections, and the OWNSI feature weighting mechanism.
Temporal Generalization Test: The study evaluates cross-day generalization by testing the models on unseen "Friday" traffic, revealing limitations in temporal transferability.
Baseline Comparison: The deep learning variants are compared against classical machine learning baselines, specifically Random Forest.

4. Results

Optimizer Performance: SCQ-SSA achieved the highest mean performance metrics among the deep learning variants: Accuracy (85.29%), Recall (88.36%), F1-score (70.53%), and ROC-AUC (0.934). ARO performed closely behind, while PSO trailed slightly.
Statistical Significance: Despite SCQ-SSA's higher mean scores, the differences between the optimizers were not statistically significant (Friedman $p \ge 0.52$ ). Furthermore, pairwise Cohen's d values did not exceed 0.26, indicating no significant practical effect size between the optimizers.
Ablation Findings: The ablation study indicated that the bidirectionality and residual connections provided a consistent, measurable benefit to the model. In contrast, the contribution of the OWNSI feature weighting was found to be less consistent and less impactful than the architectural choices.
Baseline Superiority: A Random Forest baseline significantly outperformed all deep learning variants, achieving 98.27% accuracy and a 95.52% F1-score.
Generalization: Temporal evaluation on unseen Friday traffic resulted in a 9–13 percentage point drop in accuracy, suggesting limited cross-day generalization capabilities for the models.

5. Significance and Claims

The paper explicitly moderates its claims regarding classification performance. It acknowledges that the proposed deep learning approach does not achieve state-of-the-art results compared to classical baselines like Random Forest, nor does it demonstrate superior generalization across time.

Instead, the primary significance of the study lies in its reproducible protocol for evaluating optimizers. The author argues that the value of the work is not in the specific classification metrics achieved by SCQ-SSA, but in the rigorous, statistically sound methodology used to compare it against established baselines. The study highlights that the choice of optimizer (SCQ-SSA vs. PSO vs. ARO) had a negligible impact on performance compared to the architectural design (Bi-LSTM-DRN) and that the proposed framework serves as a template for future, more rigorous evaluations in the field of network intrusion detection.