🤖 machine learning

Building an Adversarial Malware Dataset by Family and Type: Generation, Evasion, and Poisoning Evaluation

This paper introduces a publicly released dataset of over 77,000 adversarial PE malware samples derived from the RawMal-TF collection, demonstrating their high evasion rates against the EMBER classifier and their significant impact in poisoning machine learning-based detection systems.

Original authors: David Košťál, Martin Jureček

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: David Košťál, Martin Jureček

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a high-stakes game of "Spot the Fake" played between two teams: the Defenders (cybersecurity experts) and the Attackers (hackers).

For years, the Defenders have used a very smart, automated referee called EMBER. This referee looks at computer programs (specifically Windows files) and decides, "This looks like a virus" or "This looks safe." The Defenders train this referee by showing it millions of examples of bad and good files.

The Attackers, however, are getting clever. They aren't just writing new viruses; they are learning how to disguise their viruses so the referee thinks they are harmless. This is called an "adversarial attack."

This paper is about a team of researchers who decided to build a massive training gym for the Defenders. They wanted to create a huge collection of "disguised" viruses to see how well the referee could spot them, and to test what happens if the referee is tricked during its training.

Here is the breakdown of their work, using simple analogies:

1. The Raw Material: The "Monster Zoo"

The researchers started with a public collection of real-world malware called RawMal-TF. Think of this as a zoo containing thousands of real, dangerous monsters (viruses).

Group A (The Family Zoo): They sorted these monsters by their "family names" (e.g., all the "Trojan" brothers, all the "Spyware" cousins).
Group B (The Type Zoo): They sorted them by their "job descriptions" (e.g., "This one steals passwords," "This one deletes files").

2. The Disguise Artists: The "Generators"

To create the adversarial dataset, the researchers didn't just copy the monsters; they hired a team of Disguise Artists (software tools called generators).

The Goal: Take a real monster and tweak it just enough so the referee (EMBER) thinks it's a harmless puppy, but the monster still does its evil work.
The Tools: They used five different artists (like MalwareTotal, MAB-Malware, and GAMMA Sections). Some artists were like painters (adding harmless paint to the file), while others were like tailors (sewing in extra fabric to hide the shape).
The Result: They created two massive new collections:
- 44,347 disguised family-labeled monsters.
- 33,596 disguised type-labeled monsters.

The Magic Trick: These artists were incredibly good. Against the standard referee (EMBER), 98% of the family-labeled monsters and 92% of the type-labeled monsters successfully fooled the referee into thinking they were safe.

3. The "Poisoned" Training Experiment

This is the most critical part of the paper. The researchers asked: "What happens if we trick the referee while it is learning?"

Imagine the referee is a student studying for a test. The researchers took their new collection of disguised monsters and mixed them into the student's study guide.

The Trap: They told the student, "These disguised monsters are actually safe." (In reality, they are still dangerous).
The Test: They only poisoned 0.5% of the study guide (a tiny amount).

The Shocking Result:

Before the poisoning, the student could catch about 74% of the disguised monsters.
After being tricked with that tiny 0.5% of lies, the student's ability to catch the monsters plummeted to just 7%.
The Lesson: A tiny amount of "poisoned" information (mislabelled data) can completely break the student's ability to do their job, even if they still seem to be doing well on regular tests.

4. The Silver Lining: "Adversarial Training"

The researchers also tested the opposite scenario. What if they mixed the disguised monsters into the study guide but correctly labeled them as "DANGEROUS"?

The Result: The student actually got better. By studying the disguises, the student learned how to spot them. This is called "adversarial training."
The Takeaway: If you show the defender the tricks the attacker is using (and label them correctly), the defender becomes much stronger.

5. The Final Gift: A Public Library

The researchers didn't just keep these results to themselves. They packaged everything—the original monsters, the disguised versions, and all the notes on how they were disguised—into a public library (a dataset).

Why? So other researchers can use this "gym" to build better referees that can't be fooled by these tricks.
The Catch: They admitted that while their tricks worked great against the standard referee (EMBER), they were less successful against the "super-referees" (commercial antivirus software used by real companies). This means the attackers still have a lot of work to do to fool the best defenses.

Summary

In short, this paper is a massive disguise kit for malware. The authors showed that:

It is very easy to trick current AI malware detectors with the right tools.
If you lie to the AI during its training (even a little bit), you can break it completely.
If you tell the truth and show the AI the disguises, you can make it much stronger.

They released this kit to the world so defenders can practice spotting these disguises before the bad guys use them for real.

Technical Summary: Building an Adversarial Malware Dataset by Family and Type

Problem Statement
The field of malware detection relies increasingly on machine learning (ML) classifiers, yet these systems are vulnerable to adversarial attacks where malicious actors modify executables to evade detection while preserving functionality. A significant gap exists in the research community: there is a lack of large-scale, publicly available datasets containing real-world malware binaries that have been systematically transformed into adversarial samples. Existing adversarial generators often require complex setups, significant computational resources, or fail to scale to meaningful dataset sizes. Furthermore, the susceptibility of ML-based detection pipelines to data poisoning attacks—where training data is contaminated with mislabeled samples—remains under-explored at scale.

Methodology
The authors constructed a comprehensive adversarial malware dataset derived from the public RawMal-TF collection, which contains real-world Portable Executable (PE) files labeled by either family (17 distinct families) or type (14 types, e.g., trojan, spyware).

Generator Selection and Deployment:
The study utilized a suite of five automated adversarial malware generators: Pesidious, MAB-Malware, AMG, GAMMA Sections (from SecML Malware), and MalwareTotal.
- Target Classifiers: Most generators targeted the EMBER (2018) classifier. To increase diversity and challenge, MalwareTotal was also adapted to target EMBER2024, a newer, more robust feature set and model.
- Infrastructure: The authors deployed these generators in a containerized environment (Docker) on an NVIDIA DGX Station A100 server. They developed a custom "harness" script to manage parallel execution, monitor for process hangs, and handle resource constraints.
- Modifications: Several generators required code adjustments to function in the Linux environment, including replacing Windows-specific packers with the cross-platform PEzor and retraining agents to support new target classifiers.
Sample Selection and Filtering:
- The pipeline processed source samples to generate adversarial variants.
- A selection algorithm (Algorithm 1) was employed to choose the single best adversarial variant per source file. The primary criterion was the EMBER score (lower is better for evasion), with a secondary constraint on file size increase (preferring a ratio $\le$ 1.5x, with a hard cap at 25 MB).
- The dataset includes samples regardless of their original classification status (malicious or benign) to ensure diversity, provided they were successfully modified.
- Pathological cases (e.g., unmodified outputs, collapsed samples where different inputs yield identical outputs) were detected and filtered out.
Evaluation Metrics:
- Evasion Performance: Measured by the reduction in EMBER scores and the percentage of samples classified as benign.
- Antivirus Evasion: Evaluated against VirusTotal (VT), specifically analyzing detection rates across all available engines and a curated "Top 10" list of commercial AVs.
- Overhead: Measured by the increase in file size.
Data Poisoning Experiments:
The authors conducted training experiments to evaluate the impact of data poisoning. They injected adversarial samples into the training set of a LightGBM classifier (trained on RawMal-TF features) under varying conditions:
- Poisoned Fraction: The percentage of the training data replaced by adversarial samples (ranging from 0.1% to 20%).
- Transferability Rate ( $\tau$ ): The proportion of injected malicious samples that were mislabelled as benign. $\tau=0$ represents adversarial training (correct labels), while $\tau=1$ represents maximum poisoning (incorrect labels).

Key Contributions

AdvMal-TF Dataset: The release of two large-scale, publicly available datasets:
- Family-Labelled: 44,347 adversarial samples.
- Type-Labelled: 33,596 adversarial samples.
- Both datasets include rich metadata, including EMBER scores, EMBER2024 scores, and VirusTotal classifications.
Scalable Generation Pipeline: A robust, containerized framework for running multiple adversarial generators in parallel, addressing reliability issues common in research-grade malware tools.
Poisoning Analysis: Empirical evidence quantifying the vulnerability of malware classifiers to small-scale data poisoning attacks and the efficacy of correctly labeled adversarial training.

Results

Evasion Rates:
- Against the EMBER classifier, the family-labelled dataset achieved a 98.35% evasion rate for originally detected samples, and the type-labelled dataset achieved 92.20%.
- GAMMA Sections and Pesidious were particularly effective against commercial AVs on VirusTotal, though evasion rates varied significantly by generator and target.
- Generators targeting EMBER2024 achieved lower success rates, highlighting the increased robustness of the newer model.
File Size Overhead: GAMMA Sections produced the largest file size increases (often >1000%), while MalwareTotal offered the best balance of score reduction to size increase.
Poisoning Impact:
- High Vulnerability: Injecting fully mislabeled samples ( $\tau=1.0$ ) representing only 0.5% of the training data increased the evasion rate of the re-trained classifier from a baseline of 26.1% to 92.8% on the family-labelled dataset.
- Adversarial Training: Conversely, injecting correctly labeled adversarial samples ( $\tau=0$ ) consistently reduced evasion rates below the baseline, acting as effective adversarial training.
- Metric Stability: F1 scores remained deceptively stable even as evasion rates collapsed, indicating that standard metrics may fail to detect poisoning-induced degradation in adversarial robustness.
Dataset Characteristics: The family-labelled dataset showed signs of overfitting (high baseline F1 but poor cross-dataset generalization), whereas the type-labelled dataset demonstrated greater diversity and more realistic baseline performance.

Significance
The paper asserts that the AdvMal-TF dataset fills a critical resource gap by providing real-world, functionally preserved adversarial binaries at scale. The authors claim this resource will facilitate future research into:

Adversarial Malware: Understanding the capabilities and limitations of current evasion techniques.
Data Poisoning: Evaluating the robustness of detection pipelines against training data contamination.
Defensive Robustness: Enabling the development of more resilient classifiers through adversarial training with correctly labeled samples.

The work highlights that while current generators struggle against advanced classifiers like EMBER2024, the availability of diverse, large-scale datasets is essential for advancing defensive strategies. The authors conclude that the quality of training data (specifically the correctness of labels) is as critical as the volume of data in determining classifier robustness.