A Comprehensive Study of Supervised Machine Learning Models for Zero-Day Attack Detection: Analyzing Performance on Imbalanced Data

Imagine the internet as a massive, bustling city where data flows like traffic on endless highways. In this city, security guards (firewalls and antivirus software) stand at every checkpoint, checking IDs to make sure no one is sneaking in. Usually, these guards have a "Wanted Poster" book filled with pictures of known criminals. If a stranger looks like someone on the list, they get stopped. But what happens when a brand-new criminal shows up, one who has never been seen before, with a face that doesn't match any picture in the book? This is the nightmare of a "zero-day" attack: a digital intruder using a brand-new trick that security systems have never encountered.

To catch these invisible thieves, scientists are turning to a special kind of detective called Machine Learning. Think of these detectives as students who study thousands of examples of "good traffic" and "bad traffic" to learn the patterns. However, there's a tricky problem in their classroom: the teacher (the data) gives them way too many examples of good traffic and very few examples of bad traffic. It's like trying to learn to spot a rare, shy animal in a forest where 99% of the creatures are just squirrels. The student might get really good at spotting squirrels but completely miss the rare animal because they barely saw any during their training. This paper dives into how to teach these digital detectives to spot those brand-new, rare threats without getting confused by the overwhelming number of normal squirrels.

---

The Great Digital Detective Race

In this study, researchers Zahra Lotfi and Mostafa Lotfi set up a massive training ground to see which machine learning model could become the ultimate zero-day attack detector. They didn't just want to know which model was the smartest; they wanted to know which one was the fastest, too. After all, in the real world, a security guard who takes an hour to check a single person's ID is useless, even if they never miss a criminal.

The researchers used a giant dataset called UNSW-NB15, which contains over 1.4 million records of network traffic. To make the test realistic, they created a scenario where the models were trained on a mix of normal traffic and known attacks, but then tested on a "zero-day" attack that they had never seen before. To make things even harder, they used a dataset that was heavily skewed, with normal traffic vastly outnumbering the attacks, just like in real life.

They tested five different "detectives" (machine learning models):

1. Decision Tree (DT): A model that asks a series of yes-or-no questions to make a decision.
2. Logistic Regression (LR): A statistical model that calculates the probability of something being an attack.
3. Random Forest (RF): A team of many Decision Trees that vote on the answer.
4. XG Boost (XGB): A powerful team that builds on its mistakes to get smarter, known for being very fast.
5. MLP (Multilayer Perceptron): A deep learning model that mimics the layers of neurons in a human brain.

The Magic of Balancing the Scales

One of the biggest hurdles the researchers tackled was the "class imbalance problem." Since there were so few attack examples compared to normal ones, the models were struggling to learn what an attack actually looked like. To fix this, the team used a technique called SMOTE (Synthetic Minority Over-sampling Technique).

Imagine you are trying to teach a child to recognize a rare blue bird, but you only have one photo of it. The child will likely forget what it looks like. SMOTE is like a magical photocopier that creates new, slightly different photos of that blue bird by blending the original photo with its neighbors. This gives the child enough examples to really understand what the bird looks like without needing to find more real birds in the wild. The researchers applied this to their data, creating synthetic attack examples to balance the training set.

The Results: Who Won the Race?

The results were a mix of "smartest" and "fastest," and the winner depended on what you valued more.

When the models were tested without the balancing trick (SMOTE), they struggled to spot the rare attacks. For instance, the Random Forest model only caught about 83.11% of the attacks, while the MLP model was even worse, catching just 25.35%.

However, when they used SMOTE to balance the data, the performance skyrocketed.

• The Smartest Detective: The Random Forest (RF) model was the clear champion in terms of accuracy. With the balanced data, it managed to catch a massive 94.38% of the zero-day attacks (Recall) with an accuracy of 98.91%. It was the most reliable at not missing the bad guys.
• The Speedster: But here's the catch. Random Forest was incredibly slow. It took 1,379.10 seconds (over 22 minutes) to process the test data. In a real cyberattack, waiting 22 minutes to stop a hacker is like waiting for a fire to burn down the whole building before calling the fire department.
• The Best All-Rounder: Enter XG Boost (XGB). While it didn't catch quite as many attacks as Random Forest (it caught 81.40%), it was lightning fast. It finished the same task in just 5.08 seconds. It also maintained a high accuracy of 98.21%.

The Verdict

The researchers concluded that while Random Forest is technically the most accurate model for finding these invisible zero-day attacks, it is too slow to be practical for real-time security systems. If you need a guard that never sleeps and never misses a beat, but takes forever to think, you might have a problem.

Instead, they crowned XG Boost as the top choice for real-world use. It offered the best balance, being "extremely fast and precise." It proved that you don't always need the absolute perfect detector; you need one that is fast enough to stop the attack before it causes damage.

The study also confirmed that the "magic photocopier" (SMOTE) was essential. Without it, the models were blind to the rare attacks. With it, even the slower models improved significantly, though the speed of the model remained the deciding factor for which one should be used in the real world.

In the end, this research shows that in the race to catch digital criminals, being the smartest isn't enough; you have to be fast, too. And sometimes, the second-smartest detective who can run a mile in seconds is the one who actually saves the day.