💻 computer science

Dissecting Detection: Feature-Class Ablation and a Deployment-Aware Evaluation Metric for Static PE Malware Classifiers

This paper evaluates static PE malware classifiers on the Kaggle windows-malware dataset using a controlled ablation of section/structural and import/API features, demonstrating that Gradient Boosting achieves superior performance and deployment efficiency while highlighting the complementary value of feature groups through a novel latency-aware metric.

Original authors: Fahad Hafeez

Published 2026-10-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Fahad Hafeez

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Every day, antivirus software faces a monumental task: scanning hundreds of millions of files to decide, in the blink of an eye, whether each one is safe or dangerous. To handle this volume without slowing down a computer, security experts rely on static analysis. Instead of running a file to see what it does—which is risky and slow—they examine the file's structure while it sits idle. For Windows computers, most programs are built in a specific format called a Portable Executable. Think of this format as a structured container with a header that describes the contents, sections that hold the actual code, and a list of external tools the program plans to use. By reading these parts, researchers can spot patterns that distinguish malicious software from legitimate applications, often catching threats before they ever execute.

A new study by Fahad Hafeez at the University of Central Lancashire takes a closer look at how these digital detectives work. The research asks two practical questions that often get overlooked in the rush to build faster algorithms. First, which parts of a file's structure actually matter most for catching malware? Second, how should we judge a detector's success when speed is just as important as accuracy? In the real world of antivirus scanning, missing a virus is far worse than falsely flagging a safe file, and every scan must finish in milliseconds. The study uses a large, public collection of over 29,000 Windows files, mixing both safe programs and known malware, to test three different machine learning models: Support Vector Machines, Random Forests, and Gradient Boosting.

The researchers broke the file information down into three groups to see which were essential. The first group covered the file's physical structure, such as the names and sizes of its internal sections. The second group tracked the external tools the file tried to use, known as imports and API calls. The third group, intended to measure the randomness of the file's raw data, was found to be inactive in this specific dataset because the data columns were too uniform to provide useful signals. This left the team with a controlled experiment comparing the structural features against the import features. They trained their models on these different combinations and measured not just how often they were right, but how quickly they made their decisions.

The results revealed a clear hierarchy of value. The structural features alone provided a surprisingly strong foundation, allowing the models to identify malware with high accuracy even without looking at the external tools a file wanted to use. However, adding the information about external tools gave the models a significant boost, particularly for the most advanced algorithm tested, Gradient Boosting. This combination achieved the highest overall performance, correctly identifying nearly every malicious file while keeping false alarms low. The study also introduced a new way to score these detectors that weighs the importance of catching every virus against the time it takes to do so. Under this metric, the Gradient Boosting model emerged as the most useful for real-world deployment, striking the best balance between catching threats and scanning quickly.

Perhaps the most revealing finding concerned which specific clues the models relied on. By analyzing the data across all three different algorithms, the researchers identified a consensus list of the most critical indicators. These top features were almost entirely related to the external tools the files tried to use, such as specific Windows libraries for handling networks, dialogs, and memory. This suggests that while the physical shape of a file is a good starting point, the specific functions a program tries to call are the most reliable signatures of malicious intent. The study concludes that for a practical, interpretable antivirus system, focusing on these external tool requests offers the clearest path to detection, provided the system can extract that information quickly enough to keep pace with the millions of files scanned daily.

Technical Summary: Dissecting Detection – Feature-Class Ablation and a Deployment-Aware Evaluation Metric for Static PE Malware Classifiers

Problem Statement

Static analysis of Portable Executable (PE) files is a cornerstone of modern malware detection, yet two critical gaps persist in the literature. First, it is often unclear which specific groups of features (e.g., structural headers vs. API imports) drive detection performance, making it difficult for practitioners to design lightweight scanners with strict memory constraints. Second, standard evaluation metrics (accuracy, precision, recall) fail to account for the operational realities of antivirus scanning, where false negatives carry high costs, false positives cause inconvenience, and per-file inference must occur within milliseconds. Existing studies often report these factors in isolation, hindering the comparison of classifiers when detection quality and latency trade off against each other.

Methodology

The study utilizes the Kaggle joebeachcapital/windows-malwares dataset, merging four source tables (PE Header, PE Section, DLL Imports, and API Functions) into a single matrix of 29,489 samples (93.6% malware prevalence).

Feature Class Definitions and Ablation

The author partitions the feature space into three candidate classes, though the final active experiment is a controlled two-class ablation due to data filtering:

S (Section/Structural): PE header and section attributes (size, flags, entropy proxies). Initially 137 features; 132 remain active after constant-column filtering.
I (Import/API): DLL import indicators and API function calls. Initially 241 features (41 DLLs + 200 APIs) remain active.
E (Entropy/Byte-Proxy): Intended to capture byte distribution statistics. However, due to constant-column filtering, this class contains 0 active features in the final modeling matrix.

The study evaluates three non-empty feature subsets: {S}, {I}, and {S, I}.

Classifiers and Configuration

Three established supervised learning algorithms are evaluated with fixed hyperparameters to isolate the effect of feature content:

Support Vector Machine (SVM): Linear kernel, $C=1.0$ .
Random Forest (RF): 100 trees, max depth 10.
Gradient Boosting (GB): Histogram-based, 100 iterations, learning rate 0.1, max depth 3.

Evaluation Framework

The paper introduces two novel methodological components:

Detection Utility Score (DUS): A latency-aware metric defined as $DUS = \frac{F_{\beta=2}}{\log_{10}(1 + \bar{t}/t_0)}$ $D U S = \frac{F _{β = 2}}{l o g _{10} ( 1 + t ˉ / t _{0} )}$ .
- $F_{\beta=2}$ : The F-measure weighted to penalize false negatives twice as heavily as false positives ( $\beta=2$ ).
- $\bar{t}$ : Mean per-sample inference time.
- $t_0$ : Reference time of 1 ms.
- This metric allows for a scalar ranking that balances detection quality against inference speed.
Cross-Classifier Feature Consensus (CCFC): A rank-aggregation score that combines permutation feature importances from SVM, RF, and GB. Features with high CCFC scores are deemed important across algorithmic families, suggesting they reflect genuine malware behavior rather than model-specific artifacts.

Statistical significance of performance differences is assessed using McNemar's test.

Key Results

1. Classifier Performance (Full Feature Set)

On the full active feature set (S + I, 373 features), Gradient Boosting achieved the strongest overall performance:

F2 Score: 0.9972 (highest among classifiers).
AUC-ROC: 0.9950.
DUS: 293.1445 (highest, driven by high F2 and low inference latency).
Random Forest achieved the highest recall (0.9996) but suffered from higher inference latency, resulting in a lower DUS (195.7006) compared to GB.
SVM was competitive but generally outperformed by RF and GB in full-set metrics.

2. Feature-Class Ablation Findings

Section/Structural Features (S): Form a strong lightweight baseline. GB trained only on S achieved an F2 of 0.9969, significantly outperforming the majority-class baseline (F2 = 0.9866).
Import/API Features (I): Alone, they performed slightly weaker than S for all classifiers (e.g., GB F2 = 0.9896).
Marginal Contributions:
- Adding S to an I-baseline provided a consistent positive gain for all classifiers (e.g., +0.0076 for GB).
- Adding I to an S-baseline provided a small gain for GB (+0.0004) and SVM, but a slight negative impact for RF.
- Conclusion: Section/structural features are the most robust standalone class, while import/API features provide complementary signal that maximizes performance in the full GB model.

3. Cross-Classifier Feature Consensus (CCFC)

The top-ranked features by CCFC were dominated by Import/API features, specifically:

mpr.dll (Network provider router)
comdlg32.dll (Common dialog library)
getversion (OS version query)
oledlg.dll (OLE dialog)
regclosekey (Registry management)
These features reflect behavioral patterns (e.g., credential harvesting, GUI mimicry, persistence) rather than static byte anomalies.

4. DUS Sensitivity

The DUS ranking (GB > SVM > RF) remained stable across $\beta$ values ranging from 1 to 3, demonstrating the metric's robustness to the specific weighting of recall in this experimental context.

Significance and Claims

The paper positions itself not as a discovery of a new dataset or a new algorithm, but as a reproducible framework and case study for evaluating static PE detectors under deployment constraints.

Practical Utility: The study demonstrates that feature-class ablation is a viable tool for designing interpretable detectors. Practitioners can start with section/structural features for a lightweight baseline and add import/API features if computational resources allow for maximum performance.
Evaluation Innovation: The Detection Utility Score (DUS) is proposed as a necessary complement to standard metrics, explicitly incorporating inference latency to guide classifier selection in time-sensitive environments like antivirus scanning.
Adversarial Insight: The CCFC analysis highlights that high-importance features (mostly DLL/API imports) are consistent across model families. The author notes that while these are valuable for detection, their consistency also makes them prime targets for adversarial manipulation (e.g., normalizing import tables), suggesting a need for robustness in future detector design.

Limitations Acknowledged by the Author

The author explicitly bounds the scope of their claims:

Dataset: Results are specific to the joebeachcapital/windows-malwares dataset and may not generalize to broader industrial telemetry or less imbalanced distributions (93.6% malware prevalence is noted as atypical for real-world endpoints).
Temporal Validity: The study does not include chronological splits or family-level holdouts; feature importance rankings may shift as malware evolves.
Inference Measurement: Latency was measured on Google Colab CPU hardware using Python/scikit-learn. These values are relative rankings within the study, not absolute guarantees for production environments (e.g., compiled C++ scanners).
Entropy Class: The entropy/byte-proxy class was inactive in the final model, meaning the ablation was effectively a two-class comparison (Structure vs. Imports) rather than a full three-way partition.