💻 computer science

Advanced Persistent Threat Detection via Adaptive Dynamic Masking and Heterogeneity-Aware Projection

This paper proposes AMH-APT, a novel APT detection framework that enhances provenance graph analysis by employing an adaptive dynamic masking strategy to preserve critical attack context and heterogeneity-aware projection to maintain distinct semantic spaces for diverse entity types, thereby outperforming existing methods in both log-level and entity-level detection tasks.

Original authors: Jiahao Liang, Xiaofeng Lu, Xinhong Hei, Silin Guo, Danna Zhu, Shibing Gu, Pengcheng Wang

Published 2026-09-24

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiahao Liang, Xiaofeng Lu, Xinhong Hei, Silin Guo, Danna Zhu, Shibing Gu, Pengcheng Wang

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the digital world, security systems are often overwhelmed by the sheer volume of data they must watch. Every second, computers generate endless streams of records detailing who accessed a file, which program ran, or where a network connection was made. For decades, defenders have tried to spot intruders by looking for specific, known bad behaviors, like a burglar picking a lock. However, a new kind of threat has emerged that does not look like a burglar at all. These are Advanced Persistent Threats, or APTs. Instead of a single, loud break-in, an APT is a slow, quiet infiltration that stretches over weeks or months. The attackers hide their malicious steps inside the normal, boring noise of daily computer operations, making it nearly impossible to tell the difference between a routine file update and a secret data theft. To catch these stealthy enemies, researchers have begun mapping the entire history of a computer's activity into a giant web of connections, known as a provenance graph. In this web, every user, file, and program is a point, and every action they take is a line connecting them. The goal is to find the tiny, suspicious patterns hidden within this massive, complex map.

For a long time, the best tools for reading these maps relied on a technique called self-supervised learning. Imagine a student trying to learn the rules of a language by reading a book with many words covered up. The student must guess the missing words based on the context of the surrounding sentences. If they guess correctly, they are learning the structure of the language. In computer security, researchers use a similar method: they hide parts of the computer activity map and ask the AI to reconstruct the missing pieces. If the AI is good at this, it has learned what "normal" behavior looks like. When it encounters a new map where the missing pieces cannot be reconstructed because the pattern is wrong, it flags that as a potential attack. This approach has been effective, but it has a flaw. The standard method for hiding parts of the map is random; it covers up words without thinking about whether they are important. In a computer system, some actions are just background noise, like a program checking the time, while others are critical links in a chain of events, like a user opening a specific file before a secret connection is made. Randomly covering up the critical links leaves the AI with a broken story, making it harder to learn the true shape of an attack.

A team of researchers at Xi'an University of Technology has developed a new way to teach these systems, one that pays close attention to what matters. They realized that not all parts of the computer activity map are created equal. Some nodes, representing specific files or programs, are central to the story, while others are just minor details. Their new method, called AMH-APT, changes the rules of the game. Instead of randomly covering up parts of the map, the system first calculates how important each piece is. It looks at how many connections a piece has and how unique its features are. If a piece is a major hub in the network or has a very distinct behavior, the system decides to leave it visible. It only hides the less important, noisy parts of the map. This ensures that when the AI tries to guess the missing information, it is working with the most critical context available, preserving the skeleton of the attack path even while it is being trained.

The researchers also tackled a second problem: the different types of things on the map. A computer system contains users, files, network connections, and processes, all of which behave differently. Previous methods tried to squeeze all these different types into a single, uniform space, which blurred their unique characteristics. It was like trying to describe a car, a bird, and a fish using only one set of rules for movement. The new approach gives each type of entity its own dedicated space to be understood. When the system hides a file, it treats it as a hidden file, not just a generic hidden object. This prevents the AI from using the label of the object to guess its contents. By keeping the distinct identities of users, files, and networks separate while still allowing them to talk to each other, the system builds a much clearer picture of what is happening.

To test if these changes actually worked, the researchers ran their new system on five different sets of real-world data, ranging from small simulated attacks to large, complex scenarios involving thousands of events. They compared their results against the previous best method, which relied on random hiding. The findings were clear. The new system consistently found more attacks while making far fewer mistakes. In one specific test involving a large dataset called CADETS, the old method incorrectly flagged nearly 2,900 normal activities as attacks, causing a lot of false alarms. The new system reduced that number to just over 1,100, cutting the false alarms by more than half while still catching the bad actors. In another test called Wget, the system's ability to correctly identify attacks improved significantly, moving from a success rate of about 94.5 percent to nearly 98.1 percent. These improvements were not just small tweaks; they represented a fundamental shift in how the system learned to distinguish between the noise of daily life and the quiet steps of a sophisticated intruder.

The success of this method lies in its ability to balance difficulty and clarity. The researchers found that simply hiding a fixed amount of data was not enough. Instead, they introduced a schedule that started with easier tasks, hiding fewer things, and gradually increased the difficulty as the system learned. This allowed the AI to first master the local patterns of normal behavior before being challenged to understand the complex, long-range connections that define an attack. By combining this careful scheduling with the strategy of protecting important nodes and respecting the unique nature of different data types, the system learned to see the forest and the trees at the same time. The result is a detection tool that is more sensitive to the subtle signs of a breach and less likely to cry wolf over harmless activity.

This work suggests that the future of cybersecurity may depend less on finding new, complex rules and more on teaching machines how to pay attention to the right things. By understanding that not all data points are equal and that different types of digital objects require different kinds of understanding, researchers have built a system that is better at spotting the invisible. The study does not claim to have solved the problem of cyber threats entirely, but it offers a powerful new lens through which to view the chaos of digital activity. It shows that by refining how we teach machines to learn from their own history, we can make them sharper, more accurate, and more reliable guardians of our digital world. The path forward involves continuing to refine these methods, ensuring they can adapt to the ever-changing tactics of those who would do harm, while keeping the false alarms low enough that human defenders can focus on the real threats.

Technical Summary: Advanced Persistent Threat Detection via Adaptive Dynamic Masking and Heterogeneity-Aware Projection (AMH-APT)

1. Problem Statement

Advanced Persistent Threats (APTs) are characterized by low-frequency, stealthy, and long-latency activities that span multiple attack stages (e.g., initial exploitation, lateral movement, data exfiltration). These attacks are difficult to detect because they are often disguised as routine system operations within massive volumes of legitimate audit logs.

While provenance graphs effectively model these attacks by representing system entities (users, processes, files, sockets) as nodes and their causal interactions as edges, existing detection methods based on masked graph representation learning face two critical limitations:

Ineffective Masking Strategies: Current methods often employ random or fixed-ratio masking. This assumes all nodes have equal training value. However, in APT scenarios, critical attack nodes (e.g., specific process executions or cross-domain connections) are infrequent but high-value, while background nodes (e.g., temporary files, routine reads) are abundant but low-value. Random masking risks obscuring critical attack context or degenerating the training task into simple background noise reconstruction.
Heterogeneous Semantic Confusion: Provenance graphs are inherently heterogeneous, containing diverse entity types with distinct semantic spaces. Existing approaches often project all entity types into a unified feature space using shared linear transformations. This compresses type-specific distinctions, weakening the model's ability to represent complex attack semantics and potentially allowing the model to infer masked attributes from the preserved node type during pre-training.

2. Methodology: AMH-APT

The paper proposes AMH-APT (Adaptive Dynamic Masking and Heterogeneity-Aware Projection for APT Detection), which extends the MAGIC framework with two primary enhancements to the representation learning phase.

A. Node-Importance-Aware Dynamic Masking

Instead of random masking, AMH-APT employs a strategy that prioritizes the preservation of critical attack context.

Importance Estimation: Node importance ( $S(v)$ ) is calculated by fusing topological connectivity (normalized in-degree and out-degree) and attribute distinctiveness (feature variance).
Inverse-Importance Sampling: The masking probability is negatively correlated with node importance. High-importance nodes (likely part of the attack chain) are preferentially retained in the context, while low-importance background nodes are more likely to be masked. This ensures the model learns from meaningful structural dependencies rather than noise.
Curriculum-Based Scheduling: The masking rate is dynamically adjusted during training using a sigmoid function, increasing from a low value (0.3) to a high value (0.7). This allows the model to first learn stable local patterns and progressively develop higher-order contextual reasoning capabilities.

B. Heterogeneity-Aware Graph Attention Encoder

To address the limitations of unified feature projection, AMH-APT modifies the Graph Attention Network (GAT) encoder.

Type-Specific Projections: Instead of a shared linear transformation, the encoder assigns independent projection matrices ( $W_{\tau}$ ) for each node type (e.g., process, file, socket). This preserves the unique semantic nuances of different entity types before message passing.
Mask-Type Encoding: During pre-training, when a node's attributes are masked, its original node type is also replaced with a distinct "mask type." This prevents the model from leveraging the true node type as a shortcut for reconstruction, thereby strengthening the pre-training objective.
Edge-Aware Attention: The attention mechanism jointly considers source node representations, target node representations, and edge attributes to capture multi-type dependencies.

C. Training and Detection

Self-Supervised Objective: The model is trained on benign graphs using two objectives: Masked Feature Reconstruction (using scaled cosine error) and Sampled Structural Reconstruction (using binary cross-entropy to predict edge existence).
Anomaly Detection: During inference, masking is omitted. The trained encoder generates embeddings for test graphs or nodes. Anomaly scores are computed using a K-Nearest Neighbors (KNN) approach, measuring the distance of a test sample from the distribution of benign training embeddings.

3. Key Contributions

Adaptive Dynamic Masking: A strategy that integrates topological degree and attribute variance to guide masking, ensuring critical attack context is preserved while reducing background noise interference.
Curriculum-Based Scheduling: A mechanism that gradually increases masking difficulty, improving the model's generalization to unseen attack behaviors.
Heterogeneity-Aware Projection: A GAT-based encoder design that utilizes type-specific projections and explicit mask-type encoding to maintain semantic distinctions and prevent type-information leakage.
Empirical Validation: Comprehensive evaluation on five public APT datasets (StreamSpot, Unicorn Wget, Trace, THEIA, CADETS) demonstrating superior performance over baselines.

4. Experimental Results

Experiments were conducted on five datasets covering both batch-level log detection and entity-level detection.

Performance Metrics: AMH-APT consistently outperformed the MAGIC baseline and other state-of-the-art methods (e.g., Log2vec, Flash, ThreatTrace) across all datasets in terms of F1-score, Precision, and Recall.
- Wget Dataset: F1-score improved from 0.9452 (MAGIC) to 0.9810 (AMH-APT).
- CADETS Dataset: F1-score improved from 0.8978 (MAGIC) to 0.9563 (AMH-APT).
False Positive Reduction: A critical finding was the significant reduction in false positives (FP) in entity-level detection, which is vital for operational security.
- TRACE: FP reduced from 849 to 184.
- THEIA: FP reduced from 522 to 263.
- CADETS: FP reduced from 2,888 to 1,141.
Ablation Studies: Removing either the dynamic masking or the heterogeneous encoder resulted in performance degradation, confirming that both modules contribute complementary improvements (sample selection vs. semantic modeling).
Visualization: t-SNE and anomaly score distributions confirmed that AMH-APT creates a more separable embedding space, clearly distinguishing benign entities from malicious ones.

5. Significance and Claims

The paper claims that AMH-APT addresses the specific challenges of APT detection in provenance graphs by aligning the self-supervised learning process with the actual distribution of attack information. By ensuring that critical attack nodes are not randomly masked and by respecting the heterogeneous nature of system entities, the model learns more discriminative representations.

The authors emphasize that the method's ability to significantly reduce false positives while maintaining high detection rates makes it particularly suitable for practical deployment in security operations centers, where alert fatigue is a major concern. The results suggest that balancing critical-context preservation with heterogeneous semantic modeling is a viable path toward more robust, unsupervised APT detection.