🤖 machine learning

Detecting Concept Drift in Evolving Malware Families Using Rule-Based Classifier Representations

This paper proposes a structural method for detecting concept drift in evolving malware families by quantifying changes in decision tree rulesets through metrics like feature importance and prediction agreement, demonstrating that these rule-based representations effectively correlate with accuracy degradation and data distribution shifts.

Original authors: Tomáš Kalný, Martin Jureček, Mark Stamp

Published 2026-04-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tomáš Kalný, Martin Jureček, Mark Stamp

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a security guard at a high-end museum. Your job is to spot "bad actors" (malware) trying to sneak in.

For years, you’ve had a handbook of "suspicious behaviors"—maybe they wear heavy coats in summer, or they carry large, empty backpacks. This handbook is your Classifier.

But here is the problem: criminals are smart. They realize you’re looking for heavy coats, so they start wearing light windbreakers instead. They realize you’re looking for backpacks, so they start hiding tools in their pockets. This is what scientists call "Concept Drift." The "concept" of what a criminal looks like is constantly shifting. If you don't update your handbook, you'll eventually let everyone in.

What is this paper about?

The researchers in this paper wanted to find a way to tell exactly when the criminals have changed their tactics, so the security guards (the AI models) know it's time to rewrite their handbook.

Instead of just looking at whether the guard is failing (which is often too late), they decided to look at the handbook itself.

The "Handbook" Method (Rule-Based Representation)

Most AI models are like "Black Boxes"—they make decisions, but they can't tell you why. If a Black Box model starts failing, you don't know if it's because the criminals changed or because the museum got busier.

The researchers used Decision Trees, which are like a clear, step-by-step handbook:

Is the person wearing a hat? $\rightarrow$ Yes $\rightarrow$ Are they carrying a bag? $\rightarrow$ Yes $\rightarrow$ STOP THEM.

By looking at these specific rules, the researchers could track how the "logic" of the security guard changes over time.

How they measured the "Drift"

They came up with several ways to measure if the handbook was becoming obsolete. Think of these like different "alarm bells":

The Feature Importance Alarm: This asks, "Are we still focusing on the right things?" If your handbook used to focus on "hats" but now everyone is wearing "sunglasses," the importance of "hats" will drop. This is a sign the rules are getting old.
The Prediction Agreement Alarm: This asks, "Would the old handbook and the new handbook agree on this person?" If the old rules say "Let them in" but the new rules say "Stop them," something has shifted.
The Coverage Alarm: This asks, "How many people do our rules actually apply to?" If your rules are so specific that they only apply to 1% of people, your handbook is no longer useful.

What did they find? (The Results)

The "Family Feud" Trick: They found that it is much easier to spot drift when you compare one specific type of criminal against another (e.g., "Thieves vs. Spies") rather than comparing "Criminals vs. Normal People." When you compare criminals to normal people, the "normal" people are so consistent that they hide the subtle changes happening in the criminal world.
The "Two-Month Rule": They discovered that checking the handbook every two months was the "sweet spot." Checking every week was too noisy (like looking for changes in the weather every minute), and waiting too long meant you'd already been robbed.
No "Magic Bullet": There wasn't one single alarm that worked perfectly for every type of criminal. Some criminals change their "hats," while others change their "bags." To be safe, you need to listen to all the alarms at once.

Why does this matter?

In the real world, cybersecurity systems are constantly being bypassed by hackers. This research provides a way for those systems to say: "Wait! Our logic is becoming outdated. We need to study these new patterns and update our rules immediately!"

It moves us from a world where we only realize we've been hacked after the damage is done, to a world where we can see the "concept" shifting and adapt before the criminals slip through the door.

Technical Summary: Detecting Concept Drift in Evolving Malware Families Using Rule-Based Classifier Representations

1. Problem Statement

Malware evolution is a continuous "arms race" where attackers use obfuscation, packing, and encryption to evade detection. This evolution leads to concept drift—a change in the statistical properties of the target variable over time. In machine learning-based malware detection, concept drift causes significant performance degradation.

A critical challenge in cybersecurity is that many existing drift detection methods rely on error-rate-based monitoring, which requires immediate access to ground-truth labels. In real-world Security Operations Centers (SOCs), labels are often delayed, making real-time detection difficult. Furthermore, there is a lack of interpretable drift detection; knowing that a model is failing is less useful than knowing how the underlying detection logic is changing.

2. Methodology

The authors propose a structural approach to drift detection by analyzing the evolution of decision tree rulesets. Instead of monitoring prediction confidence or error rates, they quantify how the "logic" of the classifier shifts between temporal windows.

A. Experimental Setup:

Dataset: EMBER2024 (PE file subset).
Feature Selection: High-dimensional features (2568) were reduced to 10 highly discriminative features using the SelectKBest method to maintain interpretability and reduce complexity.
Classification Settings:
- Family-vs-Benign (FvB): Distinguishing a specific malware family from benign software.
- Family-vs-Family (FvF): Distinguishing between two closely related malware families (designed to highlight subtle evolutionary changes).
Windowing Strategies: The study compared fixed-interval windowing (e.g., 2-month intervals) against clustering-based windowing (using DBSCAN and KMeans to identify malware "campaigns").

B. Proposed Drift Metrics:
The authors introduced several metrics to quantify changes in the ruleset ( $R_t$ ):

Feature Importance Drift: Measures changes in the importance of features within the ruleset using Cosine Similarity, Euclidean ( $L_2$ ), Manhattan ( $L_1$ ), and Pearson Correlation (to capture changes in relative feature ranking).
Prediction Agreement: The fraction of samples where the ruleset from window $t$ and $t-1$ produce the same prediction.
Activation Stability: The cosine similarity of per-rule activation rates across windows.
Coverage Metrics: Changes in the average number of samples covered by the rules.

C. Evaluation Ground Truth:
To validate the metrics, they correlated them against two indicators:

Accuracy Degradation: The drop in model accuracy in the subsequent window.
Data Distribution Shift: Statistical shifts in the input data (measured via Wasserstein distance, Kolmogorov-Smirnov tests, and Domain Classifier AUC).

3. Key Contributions

Structural Drift Detection: A novel framework that uses the internal logic (rulesets) of a classifier as a proxy for drift, providing an interpretable way to see which features are driving the change.
Multi-Metric Approach: Demonstrates that no single metric is a "silver bullet"; different metrics capture different aspects of drift (e.g., feature ranking vs. decision boundary shifts).
Comparative Analysis of Settings: Proves that the Family-vs-Family (FvF) setting is a much more sensitive and reliable environment for detecting drift than the Family-vs-Benign setting.
Generalizability: Shows that these rule-based structural metrics are not limited to decision trees but also work with other rule-based models like RIPPER.

4. Results

Optimal Configuration: The most reliable configuration for detecting drift is fixed 2-month windowing in the Family-vs-Family (FvF) setting, using Feature Pearson Correlation. This was the only setup where all malware family pairs produced positive correlations between drift and accuracy loss.
FvF vs. FvB: Drift signals are significantly more pronounced in FvF settings. In FvB settings, the stable characteristics of benign samples often "mask" the subtle evolutionary changes in malware.
Clustering Performance: Clustering-based windowing (DBSCAN/KMeans) performed worse than fixed-interval windowing, often producing inconsistent or weak correlations.
Comparison to Baselines:
- Compared to Transcendent (a conformal evaluation framework), the proposed method is complementary. Transcendent is better at detecting sample-level deviations, while the proposed method is better at detecting decision-boundary shifts.
- The decision tree approach outperformed the RIPPER baseline in terms of stability and mean correlation.

5. Significance

This research moves malware detection from "black-box" monitoring toward explainable security. By providing metrics that correlate with both accuracy loss and data shifts, the method allows security analysts to understand the nature of the evolution (e.g., "the model is failing because the importance of entropy-based features has shifted"). This supports the development of more adaptive, trustworthy, and interpretable cybersecurity systems that can signal the need for retraining before catastrophic accuracy failure occurs.