💻 computer science

Quantitative Evaluation of APT Attack and Defense Effectiveness based on Variable Fuzzy Sets

This paper proposes the APT-VFSEA model, which utilizes variable fuzzy sets to quantitatively evaluate the effectiveness of Advanced Persistent Threat (APT) attacks and defenses by addressing their inherent uncertainty and complexity, with experimental validation demonstrating its accuracy against real-world scenarios.

Original authors: Xiang Long, Jingyuan He, Haili Sun, Lansheng Han

Published 2026-07-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xiang Long, Jingyuan He, Haili Sun, Lansheng Han

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet as a giant, bustling city. Usually, the security guards just worry about pickpockets (viruses) or people trying to smash windows (denial-of-service attacks). But then, a new kind of criminal gang arrived: the Advanced Persistent Threats (APTs). These aren't just random vandals; they are like master spies who have been hired to break into a specific bank vault. They don't just kick the door down; they spend weeks studying the blueprints, picking the lock, hiding in the walls, and stealing secrets without anyone noticing. They are tricky, complex, and incredibly hard to catch.

The problem? The old ways of measuring security were like trying to weigh a ghost. Traditional math models are too rigid. They try to say, "This attack is exactly 50% bad," or "This defense is 100% good." But in the messy, fuzzy world of APTs, nothing is that clear-cut. Sometimes a defense works perfectly in one scenario but fails in another. Sometimes the data is missing or vague. The authors of this paper argue that trying to force these fuzzy, uncertain events into rigid boxes just doesn't work.

So, the researchers from Huazhong University of Science and Technology and Wuhan Textile University decided to build a new kind of ruler. They called it APT-VFSEA.

Think of this new ruler not as a straight line, but as a shape-shifting net. In the old days, if you tried to catch a slippery fish (an APT attack) with a rigid box, it would just wiggle out. But this new "Variable Fuzzy Set" net is flexible. It understands that the "size" of an attack or the "strength" of a defense isn't a single number. Instead, it's a range of possibilities. It asks, "How much does this action belong to the 'very dangerous' category? How much does it belong to the 'moderately dangerous' category?" It calculates a score based on how well the action fits into different levels of danger, rather than forcing it into just one.

To test if their new net actually worked, the team set up a digital playground—a simulated university campus network. They invited a "red team" (the attackers) to act like the famous APT32 gang (also known as OceanLotus), using a specific weak spot in software called CVE-2017-11882. The attackers tried to trick the system with a fake document, sneak in a Trojan horse, and hide their tracks. Meanwhile, the "blue team" (the defenders) tried to stop them using tools like intrusion detectors, log collectors (Sysmon), firewalls, and antivirus software.

The researchers didn't just guess who won; they ran the numbers through their new flexible model. Here is what they found:

The attackers' first move, exploiting the vulnerability, scored a high danger level, averaging around 3.68 on a scale of 1 to 5.
The defenders' first line of defense (intrusion detection and log collection) scored lower, around 2.79 and 3.50 respectively. This matched reality: the attackers got in because these defenses weren't strong enough to stop the initial breach.
However, when the defenders used a firewall to block the specific port the attackers were using, the score jumped to 3.62.
The real hero turned out to be the antivirus software, which scored the highest at 3.93. It was the only thing that could actually find and delete the hidden files after the others failed.

The model successfully predicted that the early defenses would fail and that the antivirus would be the most effective tool, matching exactly what happened in their simulation. The authors suggest that this method is better than older ways because it handles the "fuzziness" of real-world data. It doesn't pretend to know everything with 100% certainty; instead, it embraces the uncertainty and gives a more reliable, stable score by looking at the data from multiple angles.

But here is the catch: this was a simulation. The team admits they only tested this on a small, virtual campus network. They didn't test it on massive industrial power grids or government secret networks. They also only looked at one type of attack (using a document vulnerability), not the super-secret "zero-day" attacks or supply chain hacks that real-world spies might use.

So, while the paper suggests that this new "shape-shifting net" is a much better way to measure the effectiveness of cyber attacks and defenses, it's not a magic bullet that solves every problem in the world yet. It's a promising new tool that works well in the lab, but the authors say we need to see if it holds up in the much bigger, messier battlefields of the real world.

Technical Summary: Quantitative Evaluation of APT Attack and Defense Effectiveness based on Variable Fuzzy Sets

Problem Statement
Advanced Persistent Threats (APT) are characterized by pertinence, complexity, and concealment, posing significantly greater risks than ordinary network attacks. Traditional modeling and analysis methods often fail to accurately describe and evaluate the effectiveness of APT attack and defense actions. Existing approaches suffer from several limitations:

Lack of Holistic Evaluation: Many models evaluate attack and defense effectiveness separately, lacking a unified framework that considers them as a whole within an attack-defense game.
Scenario Insensitivity: Existing models often fail to account for efficiency changes of the same actions under different scenarios.
Handling Uncertainty: APT evaluation involves significant uncertainty in both evaluation metrics and data. Deterministic methods and classical fuzzy comprehensive evaluation methods struggle to handle this uncertainty objectively, often relying on subjective human judgment or failing to utilize interval-based standards effectively.

Methodology
The paper proposes a new framework named APT-VFSEA (APT-Variable Fuzzy Sets Efficiency Assessment) to quantitatively evaluate the effectiveness of APT attack and defense actions. The methodology consists of three main components:

Unified Evaluation Criteria and Weighting:
- A flexible metric selection method is designed to be applicable to both attack and defense scenarios.
- Weight Calculation: The paper employs a linear combination of subjective and objective weighting.
  - Analytic Hierarchy Process (AHP): Used to calculate subjective weights based on expert judgment and the hierarchical structure of indicators (features, sub-features, metrics).
  - Entropy Weight Method (EWM): Used to calculate objective weights based on the information entropy of the data, reflecting the amount of information provided by each indicator.
  - The final weight is a linear combination: $w_i = \alpha \cdot w_A(i) + \beta \cdot w_E(i)$ , where $\alpha$ and $\beta$ are coefficients summing to 1.
Variable Fuzzy Set (VFS) Model Construction:
- The core of the model is the Variable Fuzzy Set, an extension of classical fuzzy sets that handles uncertainty by allowing model parameters to vary.
- Inputs: The model takes a sample matrix ( $X$ ) of evaluation data, a weight vector ( $W$ ), and an evaluation standard interval matrix ( $S_{ab}$ ) divided into $c$ levels.
- Mechanism: Instead of fixed cluster centers, the model uses interval-based cluster centers and variable range intervals ( $S_{a'b'}$ ). It calculates the relative membership degree of samples to different categories using generalized metric weight distances.
- Optimization: The model solves an objective function to determine optimal membership degrees, cluster centers, and weights. It utilizes variable parameters ( $\alpha$ and $p$ ) for the optimization criterion and distance metric (e.g., least square, Manhattan, Euclidean), allowing for multiple model combinations to enhance reliability.
Evaluation Algorithm:
- The algorithm normalizes data, calculates relative membership vectors for different model parameter combinations, and derives comprehensive relative membership degrees.
- It computes an eigenvalue for each sample based on the weighted average of membership degrees across levels.
- The final effectiveness score is the average of results obtained from different parameter combinations, ensuring stability and reducing the impact of specific parameter choices.

Key Contributions

Unified Framework: The paper proposes a method suitable for evaluating both attack and defense actions within the same scenario, addressing the lack of comparability in existing models.
Variable Fuzzy Set Application: It introduces Variable Fuzzy Sets to APT effectiveness evaluation to handle uncertainty in both metrics and data. This approach overcomes the limitations of deterministic methods and classical fuzzy comprehensive evaluation, particularly when metric standards are intervals.
Flexible Weighting: The integration of AHP and EWM provides a balanced approach to weight calculation, combining expert knowledge with data-driven objectivity.
Robustness: By combining results from four different model parameter configurations ( $\alpha \in \{1,2\}, p \in \{1,2\}$ ), the method enhances the reliability, consistency, and stability of the evaluation results.

Experimental Results
The authors validated the APT-VFSEA model using a simulated APT attack scenario involving the APT32 (OceanLotus) organization exploiting the CVE-2017-11882 vulnerability in a university network environment.

Scenario: The experiment simulated six specific actions: three attack actions (vulnerability exploitation, Trojan deployment) and three defense actions (IDS, Sysmon log collection, firewall blocking, antivirus scanning).
Findings:
- The model assigned effectiveness scores ranging from 1 to 5.
- Attack Effectiveness: Vulnerability exploitation scored ~3.68, and Trojan deployment scored ~3.31.
- Defense Effectiveness: IDS and Sysmon scored lower (~~2.79 and ~3.50 respectively) in preventing initial intrusion, consistent with the simulation where the attacker succeeded initially. However, the firewall and antivirus actions scored higher (~~3.62 and ~3.93), reflecting their success in intercepting communication and removing malicious files.
- Validation: The quantitative results aligned with the actual observed outcomes of the simulation (e.g., the failure of initial detection tools versus the success of containment and removal tools).
- Fluctuation: The evaluation results showed small fluctuation ranges (e.g., 3.59–3.74 for the vulnerability exploit), demonstrating the model's stability.

Significance and Claims
The paper claims that the APT-VFSEA model offers a reliable and accurate method for quantitatively evaluating APT attack and defense effectiveness. Its significance lies in:

Objective Handling of Uncertainty: It effectively manages the uncertainty inherent in APT scenarios without relying solely on subjective judgment.
Data Support for Strategy: The evaluation results can provide data support for reviewing past APT events, determining attack consequences, and formulating future defense strategies.
Applicability: The method is presented as having better operability, reliability, and consistency compared to existing deterministic or classical fuzzy methods.

Limitations
The authors acknowledge that the current research is limited to:

Simulating a specific document vulnerability attack chain (CVE-2017-11882) by APT32, without verifying multi-type APT groups using zero-day vulnerabilities, supply chain attacks, or other intrusion methods.
A small-to-medium-sized campus virtual network environment, lacking verification on large-scale industrial control networks or government intranets.

These limitations are noted as areas for future research.

Technical Summary: Quantitative Evaluation of APT Attack and Defense Effectiveness based on Variable Fuzzy Sets

More like this