Comparison of Multiple Classifiers for Android Malware Detection with Emphasis on Feature Insights Using CICMalDroid 2020 Dataset

Technical Summary: Comparison of Multiple Classifiers for Android Malware Detection with Emphasis on Feature Insights Using CICMalDroid 2020 Dataset

Problem Statement
The rapid release cycles of public app stores allow Android malware to spread faster than traditional signature-based scanners can react. While early detection methods relied on static signals (manifests, permissions) or dynamic analysis (system calls), the field has shifted toward large feature sets and learned representations. However, there remains a need for trustworthy detectors that utilize comprehensive datasets, rigorous evaluation, and interpretable decision drivers to avoid the pitfalls of manual signatures and heavy sandboxing.

Methodology
The study utilized the CICMalDroid 2020 dataset, comprising 17,341 Android applications collected between 2017 and 2018. The dataset includes five classes: Benign, Adware, Banking malware, SMS malware, and Riskware.

• Feature Extraction: The authors constructed a hybrid feature vector by merging static and dynamic data.
  • Static Features: Derived from JSON logs covering intents, permissions, services, file counts, obfuscation, and method tags.
  • Dynamic Features: Captured via CopperDroid virtual machine introspection, including system calls, binder calls, and composite behaviors (e.g., file system access, network read/write).
  • Processing: Initial vectors were massive (50,621 static, 470 dynamic). After variance thresholding (0.1) and handling missing values, the vectors were reduced to 301 static and 263 dynamic features, concatenated into a 564-dimensional hybrid vector, and L2-normalized.
• Experimental Design:
  • Data Split: A stratified 70% training and 30% testing split.
  • Preprocessing Schemes: Three approaches were evaluated:
    1. Original: Raw hybrid features.
    2. PCA: Principal Component Analysis retaining 95% variance.
    3. LDA: Linear Discriminant Analysis projecting onto class-separating axes.
  • Classifiers: Seven models were tested: XGBoost, CatBoost, Random Forest, Extra Trees, HistGradientBoosting, K-Nearest Neighbors (KNN), and Support Vector Machines (SVM).
  • Evaluation Metrics: Accuracy, Precision, Recall, F1-score, and confusion matrices.

Key Contributions

1. Dataset Utilization: Assembled and processed a balanced hybrid dataset of 17,000+ apps with rich metadata and behavior logs across multiple malware families.
2. Exploratory Analysis: Conducted extensive analysis including class distribution, correlation heatmaps, and visualizations (LDA and t-SNE) to understand feature relationships and class separability.
3. Comprehensive Benchmarking: Evaluated seven classifiers under three distinct preprocessing schemes, establishing performance baselines.
4. Interpretability: Employed a depth-two surrogate decision tree to identify and summarize the primary drivers of model decisions.

Results

• Classifier Performance:
  • Original Features: Gradient boosting methods outperformed all others. XGBoost achieved the highest performance with 0.9747 accuracy, 0.9703 precision, 0.9731 recall, and 0.9716 F1. HistGradientBoosting followed closely (0.9741 accuracy, 0.9708 F1). CatBoost and Random Forest were slightly lower but competitive (~0.968 accuracy). KNN and SVM lagged significantly (accuracy < 0.89).
  • PCA: Dimensionality reduction via PCA degraded performance across all models. XGBoost accuracy dropped to 0.9164, and F1 to 0.8988, suggesting that useful signal, particularly from dynamic features, was lost.
  • LDA: LDA maintained mid-90s accuracy (e.g., XGBoost at 0.9221) and clarified separable clusters in projections but did not match the performance of the original feature set.
• Class-Specific Insights: SMS malware (the largest class) achieved near-perfect F1 scores across models. Adware and Benign classes showed lower F1 scores, particularly after dimensionality reduction.
• Feature Importance: A surrogate tree analysis revealed that package name, main activity, and target SDK were the key drivers for decision-making.

Significance and Claims
The paper establishes high-fidelity supervised baselines for Android malware detection. It claims that rich hybrid features paired with gradient boosting (specifically XGBoost) offer a practical and interpretable foundation for deployment. The study highlights that while dimensionality reduction (PCA/LDA) can reveal distinct patterns, it often sacrifices detection accuracy by removing critical signal. Furthermore, the identification of metadata features (like package names) as primary drivers underscores the need for caution regarding potential spoofing by attackers. The authors conclude that their findings provide a robust, interpretable baseline, suggesting future work should focus on dataset refreshes and exploring semi-supervised or transformer-based models.