Proactive Ransomware Family Detection from Pre-Encryption API Behavior Using ANN–LightGBM Stacking

Technical Summary: Proactive Ransomware Family Detection from Pre-Encryption API Behavior Using ANN–LightGBM Stacking

Problem Statement
Ransomware defenses are most effective when malicious behavior is identified before bulk file encryption begins. However, early detection is challenging because ransomware often performs preparatory actions (e.g., querying host configuration, enumerating processes, inspecting memory) that are individually legitimate. Traditional signature-based controls fail against obfuscated or polymorphic variants, necessitating behavioral detection. This study addresses the specific challenge of classifying ransomware families from benign applications using a compact, low-dimensional representation of pre-encryption evasion-related behaviors, while investigating whether deep learning architectures offer advantages over tree-based ensembles for such small feature vectors.

Methodology
The research utilizes a curated dataset of 3,432 executable samples comprising five ransomware families (Reveton, Locky, TeslaCrypt, Yakes, Cerber) and a benign class. Each sample is represented by a 23-dimensional binary vector indicating the presence or absence of specific Windows API calls related to evasion (e.g., ReadProcessMemory, NtEnumerateKey, GetAdaptersAddresses).

The experimental protocol involved three primary evaluation settings:

1. Standalone Neural Classifiers: Five architectures were evaluated: Artificial Neural Networks (ANN), Convolutional Neural Networks (CNN), Long Short-Term Memory (LSTM), Bidirectional LSTM (BiLSTM), and Gated Recurrent Units (GRU).
2. Feature Selection: A SelectKBest method was applied to retain the top 20 features to assess if removing weak indicators improved generalization.
3. Heterogeneous Stacking: A proposed ensemble model was constructed using the ANN as the base learner and LightGBM (a gradient-boosting framework) as the meta-classifier. This design leverages the ANN's ability to model smooth nonlinear interactions and LightGBM's capacity for sharp, piecewise decision boundaries.

Evaluation was conducted using an 80:20 hold-out split and five-fold cross-validation. Performance was measured via accuracy, precision, recall, and F1-score, with specific attention to class-wise confusion.

Key Contributions

• Unified Evaluation of Deep Architectures: The study provides a comparative analysis of sequence-oriented deep models (LSTM, GRU, BiLSTM) against standard feed-forward networks (ANN, CNN) on a compact, non-sequential binary feature vector, questioning the necessity of recurrent architectures for this specific data modality.
• Feature Selection Analysis: It investigates the impact of reducing the feature space from 23 to 23 indicators, finding that feature selection benefits are architecture-dependent and not universally positive.
• Heterogeneous Stacking Strategy: The paper proposes and validates an ANN–LightGBM stacking approach that combines neural representation learning with tree-based boosting to improve decision boundaries without requiring high-dimensional sequence data.
• Critical Validity Assessment: The study explicitly outlines threats to validity, including the lack of external validation, the use of binary abstractions that discard API ordering and timing, and the absence of testing against unseen ransomware families.

Results

• Standalone Performance: In the hold-out setting, the ANN achieved the highest accuracy (94.32%) and F1-score (94.27%), outperforming the GRU (93.45%) and CNN (91.27%). Under five-fold cross-validation, the GRU demonstrated the most stability (90.85% accuracy), suggesting the hold-out results were sensitive to partition composition.
• Feature Selection Impact: Applying SelectKBest (retaining 20 features) slightly improved the ANN's accuracy to 94.47% but degraded the performance of recurrent models (LSTM and GRU), indicating that recurrent architectures may rely on specific feature interactions that are lost during reduction.
• Stacked Model Performance: The proposed ANN–LightGBM stack achieved the best aggregate performance with 95.20% accuracy, 95.35% precision, 95.20% recall, and a 95.19% F1-score. This represents a modest but consistent improvement (approx. 0.88 percentage points) over the best standalone ANN.
• Class-wise Performance: The model achieved strong F1-scores across families (0.92–0.99). The highest performance was observed for Cerber (0.99), while the lowest were for Yakes (0.92) and the Benign class (0.93). The confusion matrix indicated that Yakes samples were frequently misclassified as Locky or Reveton, and some benign samples were flagged as ransomware.

Significance and Claims
The paper claims that low-dimensional, pre-encryption behavioral features are sufficient to support accurate multiclass detection of ransomware families. It posits that heterogeneous stacking can effectively refine decision boundaries beyond what a single neural classifier can achieve, even with limited feature sets.

However, the authors maintain a modest stance regarding operational deployment. They explicitly state that the results do not establish resilience to unseen ransomware families or real-time performance. The study concludes that while the approach is competitive with recent literature, it requires external validation, temporal testing, and evaluation against concept drift and adversarial evasion before it can be considered for operational use. The primary value of the work lies in demonstrating that a lightweight feature representation combined with a heterogeneous ensemble can yield high detection accuracy without the computational overhead of complex sequence modeling or graph construction.