🤖 machine learning

Categorical Robustness Assessment for Machine Learning based Network Intrusion Detection Systems

This paper evaluates the adversarial robustness of CNN, LSTM, and Random Forest classifiers on the ACI-IoT-2023 dataset, revealing that while Random Forest achieves superior baseline accuracy, it collapses under minimal perturbations, whereas CNNs demonstrate significantly greater resilience and graceful degradation, leading to a recommendation for CNN-based architectures in adversarial network intrusion detection environments.

Original authors: Mayank Raj, Nathaniel D. Bastian, Lance Fiondella, Gokhan Kul

Published 2026-06-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mayank Raj, Nathaniel D. Bastian, Lance Fiondella, Gokhan Kul

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a security guard to watch over a busy airport. You have three candidates:

The "Perfect" Scorekeeper (Random Forest): This guard has a perfect record in training. They can spot a fake ID or a suspicious bag 99.98% of the time. They seem like the obvious choice.
The "Smooth Talker" (CNN): This guard is also very good, scoring about 99% in training. They are known for being adaptable and seeing the "big picture."
The "Memory" Guard (LSTM): This guard is also excellent, scoring around 99.3% in training. They are good at remembering sequences of events.

The paper asks a simple question: What happens when a clever thief tries to trick these guards?

The researchers didn't just ask the guards to look at normal bags; they gave them "magic tricks" (called adversarial attacks). These are tiny, almost invisible changes to the data—like a thief wearing a hat that looks normal to the eye but confuses the guard's brain.

Here is what happened when the "thieves" showed up:

1. The "Perfect" Guard Crumbled

The Scorekeeper (Random Forest) was the shocker. Even though they had the highest score in training, the moment a tiny trick was used (a very small change in the data), their performance collapsed.

The Analogy: Imagine a house built with a rigid, blocky wall. If you push it just a tiny bit in the exact wrong spot, the whole thing falls over. The Scorekeeper's decision-making is like that rigid wall. A tiny nudge makes them think a "Benign" (safe) bag is a "Bomb," or vice versa. Their accuracy dropped from 99.98% to just 26.8% with the smallest trick.
The Lesson: Just because someone has a perfect test score doesn't mean they can handle a real-world trick.

2. The "Smooth Talker" Held Their Ground

The CNN (Convolutional Neural Network) was the hero of the story. When the same tiny tricks were used, they only lost a little bit of their accuracy.

The Analogy: Imagine the CNN guard is like a flexible rubber sheet. If you poke it, it bends and wobbles, but it doesn't snap. It absorbs the small changes and keeps doing its job. Even when the tricks got bigger and more aggressive, the CNN guard degraded slowly and gracefully, rather than falling apart instantly.
The Result: The CNN kept 95.5% accuracy with small tricks, while the Scorekeeper failed immediately.

3. The "Memory" Guard Was in the Middle

The LSTM guard did okay. They were more flexible than the rigid Scorekeeper but not quite as steady as the CNN. They held their ground better than the Scorekeeper but eventually struggled when the tricks got very strong.

The Big Surprise: The "False Champion" Problem

The paper introduces a concept called the "False Champion Problem."
In the past, if you wanted the best security system, you picked the one with the highest accuracy score. This paper says that is dangerous.

The "Champion" (Random Forest) looked like the best choice because of their high score.
But in a real fight against a clever attacker, they were the weakest.
The "Runner-up" (CNN) was actually the strongest fighter.

Why Some Guards Failed at Specific Tasks

The researchers also looked at specific types of "thieves" (attack types):

The Rare Thieves: Some attacks, like "ARP Spoofing," were so rare in the training data that none of the guards could spot them well to begin with. It's like trying to teach a guard to spot a unicorn when you've only shown them horses.
The Common Thieves: Attacks like "Port Scans" were very common. The guards were great at spotting these until the "magic tricks" were used.
The "Reconnaissance" Thieves: These are the spies who look around before attacking. The paper found that all the guards struggled significantly when these specific spies were disguised with tricks.

The Final Verdict for Security Teams

If you are building a security system for the real world (like protecting IoT devices), the paper gives this advice:

Don't just look at the test scores. A high score can be a lie if the system is brittle.
Choose the "Rubber Sheet" (CNN). Even though it might have a slightly lower score in a perfect world, it is much harder to trick. It handles pressure better and doesn't crash instantly.
Be careful with rare attacks. If an attack type is very rare, no amount of training might make the system perfect at spotting it, especially if a thief tries to hide it.

In short: The paper proves that in the world of computer security, flexibility is more important than a perfect score. The model that looks perfect on paper might be the first to break when the real trouble starts.

Technical Summary: Categorical Robustness Assessment for Machine Learning based Network Intrusion Detection Systems

Problem Statement

Machine Learning (ML) based Network Intrusion Detection Systems (NIDS) have achieved near-perfect classification accuracy on benchmark datasets, yet they harbor a critical vulnerability: catastrophic failure under adversarial perturbations. While prior work has established that ML models can be manipulated via carefully crafted perturbations to network traffic data, there is a lack of systematic, cross-architecture comparison under controlled conditions. Specifically, the relationship between baseline accuracy and adversarial robustness remains poorly understood, leaving practitioners without clear guidance on which models to deploy in contested environments. This paper introduces the "False Champion Problem," where models with the highest baseline accuracy may suffer the most severe degradation under adversarial attack, rendering conventional evaluation paradigms misleading for security-critical applications.

Methodology

The study employs a multi-dimensional adversarial robustness evaluation framework to assess three fundamentally distinct ML architectures:

Convolutional Neural Network (CNN): A deep fully-connected architecture with convolutional-style regularization (Batch Normalization, Dropout).
Long Short-Term Memory (LSTM) Network: A sequential model treating network flows as single-timestep sequences to leverage gating mechanisms for feature interaction.
Random Forest (RF): A gradient-free ensemble of decision trees using axis-aligned recursive feature partitioning.

Dataset:
The evaluation utilizes the ACI-IoT-2023 dataset, comprising over 1.23 million labeled network flow samples across 12 attack classes (grouped into 5 categories: Benign, Reconnaissance, DoS, Brute Force, Spoofing). The dataset reflects real-world class imbalance, with Port Scan comprising 35.8% of samples and ARP Spoofing representing less than 0.01%.

Attack Models:

White-Box Attacks: Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD) were applied to the differentiable neural networks (CNN, LSTM) with perturbation budgets ( $\epsilon$ ) of 0.01, 0.05, and 0.1.
Transfer Attacks: Adversarial examples generated against the CNN were used to attack the non-differentiable Random Forest, simulating a gray-box scenario.
Additional Assessments: The framework includes CLEVER score computation (certified robustness), perturbability analysis, distribution shift analysis (covariate and label shift), and diffusion-based reconstruction to test generalization.

Key Contributions

Definition and Validation of the "False Champion Problem": The paper formally defines and empirically validates the inverse relationship between baseline accuracy and adversarial robustness in NIDS. It demonstrates that high clean-data accuracy is not a reliable indicator of security resilience.
Architectural Analysis of Robustness: The study links specific architectural properties to robustness outcomes. It identifies that the axis-aligned, threshold-based decision boundaries of tree-based ensembles (RF) create discontinuous surfaces that are easily exploited by minimal perturbations. Conversely, neural architectures (CNN, LSTM) learn distributed, continuous representations with smoother decision boundaries, conferring inherent regularization and greater robustness.
Categorical Vulnerability Mapping: The research reveals that adversarial vulnerability is not uniform; it varies sharply across attack categories. Reconnaissance attacks (e.g., Ping Sweep, OS Scan) exhibit universal vulnerability across all architectures, while Brute Force attacks remain relatively stable.
Comprehensive Evaluation Framework: The authors introduce a multi-dimensional assessment protocol integrating gradient-based attacks, certified metrics (CLEVER), perturbability scoring, and distribution shift analysis, establishing a new standard for evaluating security-critical ML systems.

Experimental Results

Baseline Performance: Random Forest achieved the highest baseline accuracy (99.98%), followed by LSTM (99.36%) and CNN (99.06%).
Adversarial Degradation:
- Random Forest: Suffered catastrophic collapse. Under the minimal FGSM perturbation ( $\epsilon = 0.01$ ), accuracy dropped by 73 percentage points (from 99.98% to 26.8%), and F1 score dropped by 88%. Performance remained flat as perturbation increased, indicating brittle decision boundaries.
- CNN: Demonstrated graceful degradation. At $\epsilon = 0.01$ , it retained 95.5% accuracy. Even at $\epsilon = 0.1$ (PGD), it maintained 28.7% accuracy, significantly outperforming RF in terms of stability.
- LSTM: Performed intermediately, retaining 85.0% accuracy at $\epsilon = 0.01$ (FGSM) but degrading more sharply than CNN under higher perturbations.
Class-Level Findings: Minority classes (e.g., ARP Spoofing, UDP Flood) were already weak at baseline due to data scarcity and became completely undetectable under attack. Reconnaissance attacks showed the lowest CLEVER scores (<1.5), indicating high vulnerability, whereas Brute Force classes showed higher robustness.
Transferability: Adversarial examples generated against the CNN successfully transferred to the Random Forest, confirming that cross-paradigm transferability is a significant risk.

Significance and Claims

The paper claims that the conventional wisdom of selecting NIDS models based solely on baseline accuracy is fundamentally flawed in adversarial environments. The primary significance of this work lies in shifting the focus from "accuracy maximization" to "robustness-aware architecture selection."

The authors conclude that CNN-based architectures are the most suitable for production NIDS deployments in adversarial threat environments due to their smooth, continuous decision surfaces and predictable, proportional degradation under attack. They recommend against deploying Random Forests in such settings, despite their superior clean-data performance, as their threshold-based decision boundaries render them catastrophically vulnerable to minimal perturbations. The study provides scenario-specific deployment guidance, suggesting CNNs for early detection and production use, while noting that reconnaissance detection requires additional hardening (e.g., adversarial training) regardless of the architecture chosen.