Technical Summary: Evaluating and Combatting the Impact of Concept Drift on Machine Learning-Based Phishing Detection Systems

1. Problem Statement

The paper addresses the vulnerability of machine learning (ML)-based phishing detection systems to concept drift. As phishing techniques evolve rapidly to bypass existing filters, the statistical distribution of phishing data changes over time ($P(x, y)$), causing models trained on historical data to degrade in performance. While ML methods have achieved high initial accuracy, they often become obsolete as attackers adapt. Existing solutions frequently rely on retraining or hybrid approaches that may not fully address the continuous evolution of attack vectors. The authors identify a gap in applying mathematical frameworks, previously successful in ransomware detection (e.g., the FeSAD framework), to the phishing domain to mitigate these drift effects without constant retraining.

2. Methodology

The authors propose a statistical framework designed to be algorithm-agnostic, capable of operating alongside standard ML classifiers to detect and compensate for concept drift. The methodology consists of the following core components:

2.1 Feature Selection

A modified genetic algorithm is employed to select high-information-gain features. This process aims to identify feature sets that are robust against concept drift, calculating drift values on training data to distinguish between acceptable and abnormal drift levels.

2.2 Drift Calculation Metrics

The framework calculates "sample drift" by combining mathematical distance metrics with algorithmic prediction probabilities.

• Distance Ratio ($dist_i$): Measures the ratio of a sample's distance to malign (phishing) samples versus benign samples using the Heterogeneous Euclidean Overlap Metric (HEOM). HEOM is chosen for its ability to handle both continuous and nominal attributes.
  • $dist_m$: Average distance to malign samples.
  • $dist_b$: Average distance to benign samples.
  • $dist_i = dist_m / dist_b$.
• Prediction Weights: Weights ($w_m, w_b$) are calculated by dividing the sample's prediction probability by the average incorrect prediction probability for that class in the training set.
• Weighted Distance: Combines the distance ratio and prediction weights to create a metric ($dw$) that increases separation between classes when the classifier is confident.

2.3 Drift Thresholds and Calibration

The system establishes thresholds to categorize drift:

• Expected Drift ($D_e$): Calculated using a calibration set to determine the normal range of drift between distributions (e.g., training vs. newer data).
• Drift Boundary ($D_{max}$): Defined as the midpoint between the average weighted distance of correct classifications and incorrect classifications. If a sample's drift exceeds $D_{max}$, it is considered a misclassification and re-evaluated.
• Abnormal Drift: Samples showing drift beyond the expected threshold but within the maximum boundary are flagged. If a user-defined number ($z$) of such samples accumulates, the system triggers a retraining alert.

2.4 Experimental Setup

• Datasets: The study utilizes phishing datasets spanning 2016 to 2024 to simulate concept drift. Approximately 10,000 samples were drawn from each dataset to ensure proportionate testing.
• Algorithms: Three classifiers were tested: Random Forest, Bayesian Network, and Multi-Layer Perceptron (MLP).
• Configuration: The framework was implemented in Java using the WEKA API. Experiments involved training on older data (e.g., 2016) and testing on newer data (e.g., 2021, 2024) to measure degradation and recovery.

3. Key Contributions

The paper claims three primary contributions:

1. Application of a Statistical Framework to Phishing: The authors adapt a mathematical framework (similar to FeSAD) from the ransomware domain to phishing detection, hypothesizing it can extend the lifespan of ML detection systems under concept drift.
2. Quantification of Concept Drift: The framework derives numeric values to quantify drift, allowing for the evaluation of dataset quality and feature robustness. It identifies which features provide the most distinguishable differences between benign and phishing URLs despite evolution.
3. Performance Improvement under Drift: The study demonstrates that combining a similarity index (HEOM) with classification algorithms can improve detection rates and reduce false positives when models are exposed to evolved phishing samples.

4. Results

The experiments yielded the following findings:

• Baseline Degradation: Without the framework, detection rates degraded significantly when testing on newer data. For instance, the Random Forest detection rate dropped from 86.3% (2016 data) to 78.4% (2021 data), with False Positive Rates (FPR) increasing from 10.8% to 16.9%.
• Framework Efficacy:
  • Random Forest: The framework consistently improved performance across all scenarios. When trained on 2016 data and tested on 2024 data, the framework maintained a detection rate of ~74.1% (compared to 56.9% without the framework) and reduced the FPR from 19.3% to 10.3%.
  • Other Algorithms: The framework showed significant improvements for Bayesian Networks and MLPs, particularly in scenarios of heavy degradation. For example, the Bayesian Network's detection rate on 2024 data improved from 56.9% to 74.2% when the framework was applied.
  • Abnormal Drift Reduction: The calibration component reduced the number of samples flagged as "abnormal" by approximately 45%, indicating the system successfully distinguished between normal evolution and true outliers.
• Dataset Quality Evaluation: The framework was used to evaluate the statistical distinguishability of three external datasets (Kaggle, Maribor, Badji Mokhtar). The Maribor dataset showed the highest drift difference between benign and phishing samples (1.17), correlating with the highest performance gains (Random Forest detection jumped from 93.9% to 97.8%).

5. Significance and Claims

The authors modestly claim that their research demonstrates the viability of using similarity indexes combined with ML classifiers to harden phishing detection systems against concept drift.

• Extended Lifespan: The framework allows detection systems to maintain consistent performance across different data distributions (time periods) without immediate retraining.
• Dataset Insight: The drift metrics provide an objective, statistical method to evaluate the quality of phishing datasets and the robustness of specific features, independent of the confidence of individual ML algorithms.
• Limitations: The paper acknowledges that while the framework improves detection rates, it does not completely solve the issue of high false-positive rates in all cases. The authors suggest that URL-based approaches alone may have inherent limitations regarding false positives that evolve over time.

In conclusion, the paper posits that mathematical frameworks can effectively mitigate the effects of concept drift in phishing detection, offering a mechanism to identify when retraining is necessary and improving the robustness of existing ML models.