📊 statistics

Machine Learning for Network Attacks Classification and Statistical Evaluation of Machine Learning for Network Attacks Classification and Adversarial Learning Methodologies for Synthetic Data Generation

This paper presents a unified multi-modal network intrusion detection system that combines machine learning for stable attack classification on a reprocessed dataset with adversarial learning methodologies to generate and statistically evaluate high-fidelity synthetic data.

Original authors: Iakovos-Christos Zarkadis, Christos Douligeris

Published 2026-03-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Iakovos-Christos Zarkadis, Christos Douligeris

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet as a massive, bustling city. In this city, there are security guards (Network Intrusion Detection Systems, or NIDS) whose job is to spot criminals (hackers) trying to break into homes or steal data.

For a long time, these guards have been trained using real-life police reports (real data) to recognize what a burglar looks like. But here's the problem: criminals are getting smarter. They are using advanced tools like "Generative AI" to disguise themselves, making it incredibly hard for the guards to tell a real criminal from a fake one.

This paper is like a report from two researchers who decided to upgrade the city's security system in two major ways: teaching the guards better and creating a "simulation city" to train them.

Here is the breakdown in simple terms:

Part 1: Training the Guards (The Classification Task)

The researchers first looked at how to make the security guards better at spotting real attacks. They gathered a huge pile of "police reports" from four different sources (CIC-IDS-2017, CIC-IoT-2023, etc.) and mixed them into one giant, unified database called UMNIDS.

The Challenge: The data was messy. Some crimes were very common (like pickpocketing), while others were rare (like a specific type of bank heist). Also, the data had lots of "noise" (outliers).
The Solution: They cleaned up the data and tried teaching the guards using different "teaching styles" (Machine Learning algorithms).
- They tried simple methods (like asking a guard to just look for specific patterns), but those failed miserably.
- They tried Ensemble Models (like XGBoost and Random Forests). Think of this as hiring a committee of experts instead of one single guard. If one expert misses something, another catches it.
The Result: The "Committee" approach was a huge success. They achieved over 96% accuracy in spotting attacks. The simple methods were like trying to stop a tank with a water pistol—they just didn't work.

Part 2: Building a "Simulation City" (The Synthetic Data Task)

Now, the researchers faced a new problem: What if we don't have enough police reports for rare crimes? Or what if we can't share real police reports because they contain private citizen data?

They decided to build a Virtual Simulation City using AI. The goal was to generate fake data that looks exactly like real data, so they could train the guards without risking real privacy.

The Contestants: They pitted different AI "architects" against each other to see who could build the best simulation:
- The Old School (Vanilla GAN): Like a painter who tries to copy a photo but keeps getting the colors wrong. It failed to create realistic data.
- The Modern Masters (CTGAN-2, Diffusion Forests, LLMs): These are like master forgers. They didn't just copy the data; they understood the rules of the city. They created fake records that were statistically indistinguishable from the real ones.
- The Privacy Guard (PATE-CTGAN): This architect was told, "Don't look at the real people too closely; just guess the general vibe." While this kept privacy very high, the resulting simulation was a bit too blurry and unrealistic for training guards.
The Test: How do you know the simulation is good?
1. The "Turing Test" for Data: They asked a computer to guess, "Is this a real record or a fake one?" The best AI models (CTGAN-2 and Diffusion Forests) were so good that the computer couldn't tell the difference (it guessed randomly, like flipping a coin).
2. The "Training Test": They trained a new security guard using only the fake data, then tested that guard on real criminals. The guard performed almost perfectly! This proves the fake data is useful.
3. The "Privacy Check": They checked if the AI was just memorizing and copying real people's data (which would be a privacy leak). The best models created new data that looked real but didn't steal anyone's identity.

The Big Takeaway

The paper concludes with two main lessons:

For catching hackers: Don't use a single, simple algorithm. Use a "committee" of powerful AI models (like XGBoost) to get the job done.
For training AI: We don't need to risk real private data to train our systems. We can use advanced AI (like Diffusion models and Conditional GANs) to create perfectly realistic, safe, fake data that teaches our security systems just as well as the real thing.

In a nutshell: The researchers built a super-smart security team and a perfect "flight simulator" for them. Now, the security team can practice on a million fake attacks without ever putting a single real person's data at risk.

1. Problem Statement

The paper addresses two critical challenges in modern Network Intrusion Detection Systems (NIDS):

Detection Stability: The need for reliable and stable machine learning models to detect sophisticated network attacks, particularly given the rise of AI-driven adversarial attacks that can perturb data to evade detection.
Data Scarcity and Privacy: The difficulty of training robust NIDS models due to imbalanced datasets (lack of records for rare attacks) and privacy concerns regarding real network traffic data. There is a need to generate high-fidelity synthetic data that preserves the statistical properties of real data without leaking sensitive information.

2. Methodology

The research is divided into two primary tasks: Attack Classification and Synthetic Data Generation & Evaluation.

A. Data Preparation (UMNIDS Dataset)

Source: The authors created a unified, multi-modal dataset called UMNIDS by reprocessing raw PCAP files from four existing datasets: CIC-IDS-2017, CIC-IoT-2023, UNSW-NB15, and CIC-DDoS-2019.
Features: The dataset contains ~170,000 observations with 88 features, including flow-level data, packet payload information, and temporal contextual features.
Target: A multi-class classification problem with 44 unique classes (43 attack types + 1 normal class), including Worms, Bots, DDoS, Infiltration, and SQL Injection.
Preprocessing:
- Removal of duplicates and label encoding.
- Statistical analysis (normality tests, outlier detection via IQR/Z-score/DBSCAN).
- Outlier Handling: Outliers were retained, and Robust Scaler was used for scaling.
- Imbalance Handling: Oversampling techniques (SMOTE, ADASYN, ENN) were applied only to the training set to prevent data leakage.
- Feature Selection: Recursive Feature Elimination (RFE), Stepwise selection, and Boruta were tested.

B. Task 1: Attack Classification

Models Tested: Baseline models (Logistic Regression, SVM) and advanced ensemble/tree-based models (XGBoost, Random Forest, Extra Trees, Decision Trees, Gradient Boosting variants).
Validation: Stratified 10-fold cross-validation with probability calibration.
Metrics: Precision, Recall, F1-Score, and ROC-AUC.

C. Task 2: Synthetic Data Generation

Generative Models Evaluated:
- GAN Variants: Vanilla-GAN, WGAN, WGAN-GP, f-GANs (KLD and Squared-Hellinger).
- Conditional Models: CTGAN-1 (basic), CTGAN-2 (with Bayesian Gaussian Mixture), PATE-CTGAN (privacy-preserving).
- Other Architectures: TVAE (Variational Auto-Encoder), XGB-DF (Diffusion Forest), and DistilGPT2 (LLM).
Training: Models were trained for 20 epochs using the Synthetic Data Vault (SDV) framework.

D. Evaluation Framework

The authors employed a rigorous, multi-dimensional evaluation strategy:

SDV Quality & Diagnostic Reports: Measured marginal distribution fidelity (Kolmogorov-Smirnov) and correlation similarity.
Distinguishability Tests: Trained binary classifiers (Random Forest, Logistic Regression) to distinguish between real and synthetic data.
Utility Tests (TRTS & TSTR):
- TRTS: Train on Real, Test on Synthetic.
- TSTR: Train on Synthetic, Test on Real.
- Compared against TRTR (Train/Real/Test/Real) baselines.
Statistical Hypothesis Testing:
- Multivariate Means: Regularized Hotelling's $T^2$ test.
- Covariance Matrices: Frobenius Norm test.
- Joint Distributions: Maximum Mean Discrepancy (MMD) test.
Divergence Measures: Jensen-Shannon, Hellinger, and Wasserstein distances.
Privacy: Nearest Neighbor Distance Record (NNDR) score to detect overfitting and data leakage.

3. Key Contributions

Unified Dataset: Creation of the UMNIDS dataset, integrating flow, payload, and temporal features from four major datasets into a single feature space.
Comprehensive Evaluation Framework: First known combination of the SDV framework with robust non-parametric statistical tests (Hotelling's $T^2$ , Frobenius, MMD) and f-divergences to evaluate synthetic network data.
Comparative Analysis: A systematic comparison of diverse generative architectures, including Diffusion Forests, LLMs, and Privacy-Preserving GANs, specifically for network intrusion contexts.
Stability Metrics: Introduction of a stability threshold (4% variance in metrics) to determine model reliability in intrusion detection.

4. Key Results

Classification Performance

Ensemble models outperformed baselines significantly.
XGBoost achieved the highest performance: 96.40% Precision and 96.29% Recall on the test set, with an F1-score of 96.33%.
Baseline models (Logistic Regression, SVM) failed, with SVM achieving near-random performance (F1 ~1.3%).

Synthetic Data Generation Performance

Top Performers: CTGAN-2, XGB-DF (Diffusion Forest), and DistilGPT2 achieved near-perfect SDV scores (Quality ~99-100%, Diagnostic 100%).
Distinguishability:
- CTGAN-2, XGB-DF, and DistilGPT2 were difficult to distinguish from real data (F1 scores ~52-54%, ROC-AUC ~64%, indicating the classifier struggles to tell them apart).
- PATE-CTGAN was easily distinguished (F1 100%, ROC-AUC 100%), indicating the privacy constraints severely degraded data utility.
Utility (TRTS/TSTR):
- Models trained on synthetic data (XGB-DF, CTGAN-2, DistilGPT2) achieved performance levels nearly identical to the TRTR baseline (Precision/Recall ~95-96%).
- PATE-CTGAN failed utility tests (Precision dropped to <1% on test sets).
Statistical Validity:
- Means: No significant difference found between real and synthetic means for any model (p-values > 0.05).
- Covariance: XGB-DF, DistilGPT2, and PATE-CTGAN showed no significant difference in covariance matrices. However, CTGAN-2 showed a statistically significant difference in covariance (p=0.0019), suggesting slight structural deviations despite high utility.
- Joint Distribution (MMD): XGB-DF and DistilGPT2 perfectly matched the joint distribution (p=1.0). CTGAN-2 was close (p=0.08). PATE-CTGAN failed (p=0.0000).
Privacy: PATE-CTGAN showed the highest privacy scores (NNDR ~0.97), while other models showed signs of overfitting (lower NNDR), though still within acceptable ranges for non-privacy-preserving models.

5. Significance and Conclusion

The paper demonstrates that ensemble learning (specifically XGBoost) provides the most stable and accurate detection for network attacks in tabular data. Regarding synthetic data, the study concludes that Conditional GANs with mixture models (CTGAN-2), Diffusion Forests (XGB-DF), and LLMs (DistilGPT2) are superior for generating synthetic network data. They successfully capture complex multivariate distributions and offer high utility for training NIDS models without significant data leakage.

Conversely, while PATE-CTGAN offers superior privacy, it currently sacrifices too much data fidelity and utility for practical NIDS training. The proposed evaluation framework provides a robust, statistically grounded method for validating synthetic data, moving beyond simple visual inspection to rigorous hypothesis testing. Future work suggests applying these methods to cross-dataset validation and exploring Bayesian mixtures for unbalanced data.