A Unified Evaluation of Learning-Based Similarity Techniques for Malware Detection

1. Problem Statement

Modern malware analysis requires identifying relationships between binary files (e.g., family attribution, clustering) rather than just binary classification (malicious vs. benign). Traditional cryptographic hashes (MD5, SHA-256) fail here because minor transformations (packing, recompilation, instruction reordering) produce completely different hashes.

While fuzzy hashing techniques (e.g., ssdeep, TLSH) and machine learning (ML) based embedding methods exist, they have historically been evaluated in isolation using disparate datasets and metrics. There is a lack of a unified framework to compare:

• Classical byte-level similarity digests.
• Learning-based embeddings derived from static PE metadata.
• The trade-offs between classification accuracy and the quality of the induced similarity space (clustering).

2. Methodology

The authors present a reproducible, unified experimental framework using the EMBER (1 million static PE files) and EmberSim datasets.

A. Data Preprocessing & Feature Engineering

• Source: Static PE metadata extracted via LIEF into JSONL format.
• Transformation: A rigorous pipeline converts nested, heterogeneous JSON structures into fixed-length numerical vectors.
  • Normalization: Excludes non-informative identifiers (hashes).
  • Flattening: Unrolls hierarchical structures (e.g., COFF headers, imports) into discrete columns.
  • Vectorization: Uses Bag-of-Words (count vectorization) for high-cardinality string features (imports, exports) with a vocabulary limit of 2,048, and z-score normalization for numeric features.

B. Models Evaluated

The study compares four distinct approaches to generate similarity representations:

1. Classical Fuzzy Hashing: ssdeep (used as a baseline).
2. Unsupervised Autoencoder (AE): A symmetric MLP (256→64→8→64→256) trained to reconstruct input features, using the 8-dimensional bottleneck as the embedding.
3. Deep Learning Classifier (Binary): A fully connected DNN (512→256→128) trained for malware/benign classification. The 128-dimensional embedding layer is extracted for similarity tasks.
4. Tree-Based Models (XGBoost):
   • Trained for binary classification and multi-class AVClass family classification.
   • Embedding Extraction: Uses the leaf-index of the 100 trees in the ensemble to create a 100-dimensional vector (samples traversing similar paths are close in space).

C. Evaluation Metrics

The study uses a unified set of metrics to evaluate both classification and similarity:

• Classification: Accuracy, Precision, Recall, F1-Score, AUC.
• Similarity/Clustering Quality:
  • Silhouette Score: Measures cluster cohesion and separation.
  • Davies–Bouldin Index (DBI): Lower is better; measures average similarity between clusters.
  • Calinski–Harabasz Index (CH): Higher is better; ratio of between-cluster to within-cluster dispersion.
  • Label Homogeneity@K: The percentage of Top-K nearest neighbors sharing the same label as the query sample.

3. Key Contributions

1. Unified Benchmarking Framework: The first side-by-side comparison of classical fuzzy hashes and diverse learning-based embeddings (AE, DNN, XGBoost) on a shared dataset using consistent metrics.
2. Systematic Feature Engineering: A robust pipeline for transforming raw PE metadata into fixed-length vectors suitable for both tree-based and neural models.
3. Comprehensive Comparative Analysis: Evaluation of unsupervised, supervised, and tree-based approaches for both binary detection and multi-class family clustering.
4. Reproducibility: The study is fully reproducible, addressing the "black box" nature of previous isolated evaluations.

4. Key Results

A. Classification Performance

• Binary Detection: XGBoost outperformed Deep Learning, achieving 97.76% accuracy and an AUC of 0.9971.
• Multi-Class (AVClass): XGBoost again led with 90.55% accuracy (vs. 80.15% for Deep Learning) and superior macro-F1 scores, indicating better generalization across rare malware families.

B. Similarity and Clustering Performance

• Label Homogeneity:
  • Autoencoders (Unsupervised): Achieved >80% label homogeneity in Top-K queries, significantly outperforming ssdeep (approx. 40%).
  • Deep Learning Embeddings: Showed superior neighborhood purity (94.51% homogeneity at K=100) compared to XGBoost leaf-index embeddings (71.69% without AVClass features).
• Clustering Metrics:
  • Deep Learning embeddings demonstrated significantly better cluster separation (higher Silhouette, lower DBI) than XGBoost embeddings, particularly for multi-class (family) clustering.
  • Critical Finding: High classification accuracy (XGBoost) does not guarantee a geometrically well-structured similarity space. XGBoost achieved the best classification but the worst clustering metrics (negative Silhouette score for AVClass), whereas Deep Learning provided a more cohesive embedding space.

C. The "Supervision Shortcut" Bias

The study revealed that when XGBoost is trained with AVClass labels as input features, it achieves near-perfect homogeneity (98%). However, this is a "shortcut" because AVClass is often a direct indicator of the family. In real-world scenarios where AVClass is unavailable, XGBoost's similarity performance degrades significantly compared to neural embeddings.

5. Significance and Conclusion

• No Single Solution: No single method dominates all dimensions. XGBoost is superior for high-accuracy classification, while Deep Learning and Autoencoders are superior for generating robust similarity embeddings and clustering.
• Hybrid Approach Required: Effective malware analysis platforms must integrate complementary techniques. Relying solely on classification accuracy to infer similarity quality is flawed.
• Limitations: The study was limited to pre-extracted static features (EMBER dataset) and could not directly compare against raw-byte fuzzy hashes (like TLSH) on raw binaries.
• Future Directions: The authors propose integrating dynamic analysis features (runtime behavior) and evaluating scalability with approximate nearest-neighbor search on raw binary corpora.

Final Takeaway: For threat hunting and malware clustering, learning-based embeddings (specifically Deep Learning and Autoencoders) provide a more structurally sound similarity space than traditional fuzzy hashes or tree-based leaf indices, even if the latter offer better raw classification accuracy.