🤖 machine learning

Adversarial Co-Evolution of Malware and Detection Models: A Bilevel Optimization Perspective

This paper proposes a robust malware detection defense framework using bilevel optimization to model the adversarial co-evolution between attackers and defenders, significantly reducing evasion rates and increasing attacker costs compared to traditional methods.

Original authors: Olha Jurečková, Martin Jureček, Matouš Kozák, Róbert Lórencz

Published 2026-04-27

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Olha Jurečková, Martin Jureček, Matouš Kozák, Róbert Lórencz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a high-stakes game of "Hide and Seek" played between a world-class Hide-and-Seeker (the Malware Detector) and a master of Disguise (the Malware).

In the past, the game was simple: the Seeker learns where people usually hide, and the Hider learns to wear a new mask. But in the digital world, this game has become incredibly intense and dangerous.

Here is a breakdown of the paper’s findings using that analogy.

1. The Problem: The "One-Trick Pony" Defense

Traditionally, when a security company discovers a new way that hackers are hiding (a new "disguise"), they update their software. This is called "One-Shot Training."

The Analogy: Imagine a security guard who sees a thief wearing a clown mask to sneak into a building. The guard says, "Okay, from now on, I won't let anyone in wearing a clown mask."

The problem? The thief is smart. As soon as they see the guard looking for clowns, they put on a pirate hat. The guard is still looking for clowns, and the thief walks right past them. This is what happened in the paper: standard defenses were being bypassed 90% of the time.

2. The Solution: The "Evolutionary Training Camp" (Bilevel Optimization)

The researchers decided that instead of just reacting to one disguise, they needed to create a "training camp" where the Seeker and the Hider evolve together in a continuous loop. They call this Bilevel Optimization.

The Analogy: Instead of a single guard, imagine a Training Simulation.

Round 1: The Hider tries to sneak in using a clown mask. The Seeker catches them.
Round 2: The Hider realizes the Seeker is watching for clowns, so they try a pirate hat. The Seeker immediately learns about the pirate hat and prepares for it.
Round 3: The Hider tries a ninja outfit. The Seeker is already one step ahead.

This isn't just a one-time update; it’s a co-evolutionary loop. The defender (the Seeker) is constantly "playing" against an attacker who is also constantly getting smarter. By the time the "real" game starts, the Seeker has already practiced against almost every trick in the book.

3. The Results: Making the Thief "Give Up"

The researchers tested this against three different "families" of digital thieves (Mokes, Strab, and DCRat). The results were dramatic:

Total Immunity: While the old way of defending let 90% of thieves through, this new "Evolutionary Training" method stopped almost all of them (reducing the success rate of thieves to nearly 0%).
The "Cost" of Hiding: In the old way, a thief could find a disguise very quickly. With the new defense, the thief had to try thousands of different disguises before finding even a tiny crack in the armor.

The Analogy: It’s like the difference between a thief finding a door left unlocked (easy) and a thief having to try 10,000 different combinations on a high-tech safe just to get a single glance inside (exhausting and expensive).

Summary in a Nutshell

The paper argues that we can't protect computers by just "patching" holes as they appear. Because hackers use AI to adapt, our defenses must also use AI to anticipate that adaptation. By treating cybersecurity as a continuous, evolving game rather than a static wall, we can create detectors that are not just strong, but "future-proof."

Technical Summary: Adversarial Co-Evolution of Malware and Detection Models

1. Problem Statement

The paper addresses a fundamental asymmetry in cybersecurity: while machine learning (ML)-based malware detectors are trained on static, historical datasets, adaptive attackers use reinforcement learning (RL) to generate new, sophisticated malware variants designed specifically to bypass these defenses.

Traditional defense mechanisms, such as one-shot adversarial training (retraining a model once on a set of adversarial examples), are insufficient. They fail to account for the iterative and strategic nature of the attacker, who can adapt their strategy as soon as a new defense is deployed. This creates a "cat-and-mouse" game where static defenses are quickly rendered obsolete by evolving evasion techniques.

2. Methodology

The authors propose a shift from static defense to a strategic co-evolutionary framework modeled through bilevel optimization.

The Attacker (Inner Problem): The study utilizes the MAB-malware framework, which treats adversarial malware generation as a Multi-Armed Bandit (MAB) problem. The attacker uses RL to discover functionality-preserving transformations (e.g., section injection, header manipulation) that maximize the probability of a malicious file being classified as benign.
The Defender (Outer Problem): The defender aims to minimize classification loss. Instead of a single training step, the defender engages in Iterative Best-Response (IBR) dynamics.
Bilevel Optimization Framework: The interaction is formulated as:
$\min_{\theta \in \Theta} \text{Loss}(f_\theta(x + \delta^*(\theta), y))$
$\text{subject to } \delta^*(\theta) \in \arg\max_{\delta \in S(x)} \text{EvasionLoss}(f_\theta(x + \delta), y)$
In this setup, the attacker finds the optimal perturbation $\delta^*$ for the current model $\theta$ , and the defender then updates $\theta$ to be robust against that specific optimal response. This cycle repeats for several iterations, effectively simulating a game-theoretic equilibrium.

Experimental Setup:
The researchers compared three approaches across three malware families (Mokes, Strab, and DCRat):

Baseline: Standard Random Forest trained on clean data.
Adversarial Training: One-shot retraining using adversarial samples.
Bilevel Optimization: The proposed iterative co-evolutionary approach.

3. Key Contributions

Game-Theoretic Modeling: Formulating malware detection not as a static classification task, but as a dynamic, iterative game between two adaptive agents.
Bilevel Optimization via IBR: Providing a practical implementation of bilevel optimization using Iterative Best-Response, which allows the defender to "exhaust" the attacker's potential strategies.
Robustness Benchmarking: Demonstrating that modeling the process of attack and defense is more critical than simply increasing the volume of training data.

4. Results

The experimental results show a massive performance gap between traditional methods and the proposed framework:

Metric	Baseline (Exp 1)	Adv. Training (Exp 2)	Bilevel (Exp 3)
Evasion Rate (ER)	High (up to 96%)	Moderate (up to 90%)	Near-Zero (0% – 1.89%)
Attacker Complexity	Low (avg. ~16 queries)	Moderate	Very High (up to 3118 queries)

Mokes Family: The bilevel approach achieved 0% evasion and perfect detection scores.
DCRat Family: While standard adversarial training failed completely (90% evasion rate), the bilevel approach neutralized the threat entirely (0% evasion).
Strab Family: The model achieved a marginal evasion rate of only 1.89%, significantly outperforming the 77% evasion rate of the baseline.
Attacker Cost: The framework significantly increased the "cost of attack." For the Strab family, the average number of queries required for a successful evasion rose by two orders of magnitude.

5. Significance

This research demonstrates that resilience in malware detection is a product of adaptation, not just data volume. By modeling the interaction as a bilevel optimization problem, the authors show that defenders can reach a practical $\epsilon$ -Nash Equilibrium, where the attacker can no longer find effective new evasion paths without an unsustainable increase in computational cost. This provides a blueprint for developing next-generation, "self-hardening" cybersecurity systems capable of withstanding evolving, RL-driven adversarial threats.