Related Families, Not Label Errors: A Case Study of a Failure Mode in Open-World Malware Evaluation on BODMAS

Technical Summary: Related Families, Not Label Errors in Open-World Malware Evaluation

Problem Statement
Open-world malware detection typically employs a "leave-one-family-out" (LOFO) protocol, where a specific malware family is excluded from training, and the detector is evaluated on its ability to flag samples from that held-out family as "novel" (out-of-distribution). This paper identifies a critical failure mode in this evaluation paradigm that is unrelated to the detector's performance or data labeling errors. Specifically, if a held-out family has a close relative (a genetically or feature-space similar family) that remains in the training set, the held-out samples are not genuinely unseen. Distance-based novelty scores, which measure how far a sample is from the training distribution, will incorrectly classify these related samples as "known" because they lie within the region covered by the relative family. Consequently, the detector is penalized with an Area Under the ROC Curve (AUROC) significantly below chance (inverting the metric), not because it failed to detect novelty, but because the evaluation premise was violated by family relatedness.

Methodology
The authors conduct a detailed case study on the BODMAS benchmark, utilizing static Portable Executable (PE) feature vectors. The study employs a rigorous, leakage-free protocol where standardization and Principal Component Analysis (PCA) are fitted exclusively on training families to prevent data leakage.

Key methodological components include:

1. Causal Intervention: The authors isolate the cause of low scores by holding out both a suspected "problematic" family and its closest relative simultaneously. If the low score is due to relatedness, the AUROC should recover to near-perfect detection when both are removed from training.
2. Audit for Label Errors: To rule out the hypothesis that low scores are caused by duplicate labels (i.e., the held-out family is actually the same as a training family), the authors performed a two-stage audit on the 53 largest families (1,378 pairs). This involved a logistic regression screen followed by a random forest two-sample test with balanced class sizes to adjudicate the top 20 suspicious pairs.
3. Metric Agnosticism: The study evaluates six unsupervised novelty scorers (Mahalanobis distance, k-NN, Isolation Forest, PCA reconstruction, Autoencoder, and One-Class SVM) and one supervised baseline (Random Forest) to determine if the failure is specific to a scoring rule or the evaluation protocol itself.
4. Taxonomy Analysis: The authors examine antivirus vendor alias lineages (specifically the berbew, qukart, and padodor lineage) to test if name-based normalization could resolve the issue.

Key Results

• The Inversion Phenomenon: Under the leakage-free protocol, the family berbew achieved an AUROC of 0.184 (significantly below the 0.5 chance line) when held out alone. This occurred because its close relative, qukart, remained in training. When both berbew and qukart were held out together, the AUROC for berbew recovered to 0.903. The other 18 families showed negligible change (<0.001).
• Cause Confirmation: The audit confirmed that berbew and qukart are distinct families with no shared feature vectors. A supervised classifier separates them with 0.752 balanced accuracy, proving the labels are correct but the families are highly proximate in feature space (cosine similarity >0.99).
• Metric Specificity: The failure is specific to unsupervised novelty scoring. Four global-model unsupervised scorers (Mahalanobis, Isolation Forest, PCA, Autoencoder) all inverted (scored below chance) on the related families. A purely local k-NN score degraded significantly but did not invert. A supervised classifier did not invert at all, confirming the issue lies in the unsupervised assumption that the held-out family is far from the entire training distribution.
• Anomaly: The family sillyp2p also scored below chance (0.428) but was well-separated from its nearest neighbor. The authors note this inversion is not explained by the single-nearest-family mechanism and remains an open question, though it also fails unsupervised metrics while recovering under supervised scoring.
• Taxonomy Limitations: In the examined lineage, padodor is separable from berbew (0.972 accuracy) despite being part of the same vendor-identified lineage. Name-based normalization would incorrectly merge separable families while failing to identify the problematic berbew/qukart pair.

Key Contributions

1. Demonstration of Relatedness-Driven Failure: A controlled demonstration that family relatedness, not label noise, can drive unsupervised novelty scores below chance, effectively penalizing detectors for correct behavior (identifying a variant of a known lineage).
2. Benchmark Audit: A comprehensive audit of BODMAS finding no duplicate labels among the most suspicious pairs, validating that the failure is structural rather than a data quality issue.
3. Protocol Diagnosis: Evidence that the failure is a property of the LOFO evaluation protocol when applied to unsupervised novelty detection, not a deficiency of a specific metric.
4. Methodological Transparency: Documentation of four method attempts that failed to beat a nearest-neighbor baseline and six specific measurement pitfalls (e.g., class imbalance in separability stats, calibration leakage) encountered and corrected during the research.

Significance and Claims
The paper argues that the remedy for this failure mode is not to remove related families from benchmarks, as variants of known lineages are operationally relevant cases practitioners must handle. Instead, the authors propose:

• Disclosure and Stratification: Benchmarks should disclose the pairwise separability of held-out families from their nearest training relatives.
• Stratified Reporting: Novelty performance should be reported separately for families with close relatives in training versus those without, rather than relying on a single pooled mean which obscures the inversion.
• Local vs. Global Scoring: The study suggests that local scoring methods (like k-NN) are more robust to this specific failure mode than global-model scorers, though they are not immune to performance degradation.

The authors explicitly frame this work as a detailed case study on the BODMAS benchmark and static PE features, rather than a general law claiming this failure is endemic across all malware benchmarks. The primary contribution is the diagnostic framework: open-world benchmarks must be checked for family relatedness before being used to rank detectors, as a single related pair can produce pathological evaluation results that mislead the community.