A Comparison of Malware Image Transformations Using Grad-CAM and Hybrid Learning Models

Technical Summary: A Comparison of Malware Image Transformations Using Grad-CAM and Hybrid Learning Models

Problem Statement
While converting malware binaries into 2-D images has proven effective for machine learning-based detection and classification, performance varies significantly depending on the specific transformation technique used. Furthermore, the explainability and interpretability of these image-based models remain largely unexplored within the malware domain. High-performing Deep Neural Networks (DNNs) often function as "black boxes," limiting analyst trust and adoption in security applications. There is a critical need to understand how different binary-to-image transformations influence both classification accuracy and the structure of explainable AI (XAI) explanations, specifically Gradient-weighted Class Activation Maps (Grad-CAM). Additionally, it is unclear whether the visual patterns generated by XAI tools (heatmaps) contain sufficient structural information to serve as a secondary representation for classification tasks.

Methodology
The research utilizes the RawMalTF dataset, comprising 17,000 malware samples across 17 distinct families. Each sample is converted into eight distinct image representations: Grayscale, Entropy Hilbert, Byteclass Hilbert, Hybrid Image Transformation (HIT), Bigram Cartesian, Bigram Polar, Spiral, and Byteclass.

The study employs a multi-faceted experimental approach:

1. Model Architecture: A MobileNetV2 Convolutional Neural Network (CNN), initialized with ImageNet weights, serves as the backbone for all deep learning experiments. The architecture includes a Global Average Pooling layer followed by a dense classification head. Training involves a two-stage process: freezing the backbone to train the head, followed by fine-tuning the upper 30% of the backbone layers.
2. Grad-CAM Analysis: Grad-CAM heatmaps are generated for every sample to visualize the regions driving classification decisions. These heatmaps are analyzed qualitatively and quantitatively.
3. Feature Extraction from XAI: The Grad-CAM heatmaps are treated as input images. Hand-crafted features are extracted from these maps, including global intensity statistics, geometric shape features (Hu moments, eccentricity), Local Binary Patterns (LBP), and Gray-Level Co-occurrence Matrices (GLCM). Classical machine learning models (Random Forest, XGBoost, SVM) are trained on these feature vectors.
4. Hybrid Pipelines:
   • CNN on Overlays: CNNs are trained directly on the Grad-CAM overlay images (original image blended with the heatmap).
   • Hybrid CNN-HOG-XGBoost: CNN embeddings (from the penultimate layer) are concatenated with Histogram of Oriented Gradients (HOG) features extracted from the overlay images, fed into an XGBoost classifier.
   • Progressive Feature Fusion: Embeddings are extracted from models trained on both original and overlay images across all eight transformations. These are concatenated into feature vectors of increasing dimensionality (256, 512, and 4096 dimensions) and classified using Random Forest.
5. XAI Evaluation: The study compares Grad-CAM against HiResCAM. It quantitatively evaluates explanations using faithfulness metrics (Deletion/Insertion AUC, drop top20pct, concentration) and stability metrics (Spearman correlation, MSE, SSIM, Top-K overlap) under input perturbations.

Key Results

• Accuracy vs. Transformation: Classification performance varies widely across transformations. The Entropy Hilbert transformation yielded the highest accuracy for a single CNN model (0.688), while Byteclass Hilbert and Spiral performed the worst.
• Accuracy vs. Explainability: A key finding is that classification accuracy and explanation faithfulness do not coincide. For instance, the Bigram Polar transformation produced the most faithful explanations (highest faithfulness score) but ranked only mid-tier in accuracy. Conversely, Grayscale achieved high accuracy but low faithfulness. Entropy Hilbert emerged as the most balanced transformation, performing strongly in both accuracy and faithfulness.
• Stability vs. Faithfulness: An inverse relationship was observed between faithfulness and stability. Transformations with high faithfulness (e.g., Bigram Polar) exhibited lower stability under noise, while highly stable transformations (e.g., Byteclass, Grayscale) showed lower faithfulness.
• Hybrid Performance: Hybrid approaches consistently outperformed CNN-only models. The CNN-HOG-XGBoost pipeline achieved a peak accuracy of 0.731 (using HIT images).
• Feature Fusion: The most significant result was achieved by combining embeddings from all 16 models (original and overlay images for all eight transformations) into a 4096-dimensional feature vector. A Random Forest classifier trained on this vector achieved a test accuracy of 0.777, exceeding the previous benchmark of 0.751 for this dataset.
• Grad-CAM vs. HiResCAM: For the MobileNetV2 architecture, Grad-CAM and HiResCAM produced effectively equivalent explanations at the final convolutional layer, with negligible pixel-level differences.
• Family Difficulty: Certain families (Noon, Androm, Agensla, Injuke) remained difficult to classify across all experiments, suggesting inherent structural similarities that are hard to distinguish using image-based features alone.

Significance and Contributions
This research provides a systematic evaluation of the trade-offs between image transformation choices, classification accuracy, and explanation quality in malware analysis. The paper demonstrates that:

1. Explanations as Data: Grad-CAM heatmaps are not merely visualization tools; they preserve meaningful family-discriminative structure and can be used as input for classification, sometimes improving performance over original images.
2. Complementary Features: Combining deep learning embeddings with classical hand-crafted features (HOG) and aggregating representations across multiple transformation types yields superior accuracy compared to single-model approaches.
3. Decoupling Metrics: Accuracy, faithfulness, and stability are distinct properties. Optimizing for one does not guarantee optimization for the others, necessitating a balanced approach for practical security applications where both correctness and robustness are required.
4. Benchmark Improvement: By leveraging a progressive feature fusion strategy, the study establishes a new state-of-the-art accuracy (0.777) for the 17-family malware dataset, surpassing previous benchmarks based on HOG features alone.

The work underscores that while no single transformation dominates all metrics, hybrid strategies and the strategic combination of multiple representations offer a viable path toward building malware classification systems that are simultaneously accurate, interpretable, and robust.