Graph Neural Networks for Provenance-Based APT Detection under a Temporal Transfer Framework: A Review

In the hidden layers of computer systems, a silent war is constantly waged. Attackers known as advanced persistent threats do not crash servers or steal data in a single, noisy burst. Instead, they move slowly, like a burglar who enters a house, waits for the family to sleep, and then quietly moves from room to room, opening drawers and copying documents over the course of days or weeks. To catch such a patient intruder, security experts look at the system's history, or provenance. This is a record of every action a computer takes: which program opened which file, which process sent a message to another, and when these events happened. By connecting these dots, experts can see the invisible path an attacker leaves behind. For years, researchers have tried to use artificial intelligence to read these histories automatically, hoping to spot the subtle patterns of an attack before it is too late.

A new review of this field, however, suggests that the current success stories may be an illusion. The researchers examined dozens of studies that claimed to have built artificial intelligence models capable of spotting these long-term intrusions with near-perfect accuracy. They found that while these models perform brilliantly in the lab, they often fail to account for the most critical factor in real-world security: time. The studies reviewed mostly tested their models on data that was already complete, allowing the artificial intelligence to see events that had not yet happened at the moment an alert should have been raised. It is a bit like grading a student on a history exam by letting them read the answer key before they sit down to write. When the researchers looked closer, they discovered that the models were not necessarily learning to recognize the attacker's behavior; they were often just memorizing the specific details of the test data or relying on information that would not be available in a live, real-time situation.

The review, conducted by a team of computer scientists, synthesized findings from twenty-nine different detection studies published between 2020 and 2026. These studies used a type of artificial intelligence called a graph neural network, which is designed to understand relationships between things, much like a map showing how cities are connected by roads. In this context, the "cities" are computer processes and files, and the "roads" are the actions connecting them. The researchers organized these studies into a clear framework to see how they handled the passage of time. They found that while nineteen of the studies used these advanced graph networks, very few actually tested how well the models worked on data from a later date. In the real world, a security system is trained on data from last month and must detect attacks happening today. The review found that most studies did not simulate this delay. Instead, they trained and tested their models on the same batch of data, often mixing up the timeline so that the model could "peek" at future events to make its decision.

One of the most significant findings was that the high accuracy numbers reported in these studies likely do not translate to real-world performance. The researchers defined two levels of testing to prove this. The first level asks if a model, once trained and frozen, can still spot attacks when it is shown new data from a later time. The second level asks if the model can adapt and improve as it sees new data. The review discovered that while five studies mentioned ways to adapt their models, none of them actually tested these adaptations on data from a later period. Furthermore, many studies used a method of labeling attacks that created a hidden trap. If the researchers marked a file as "bad" because it was connected to a known bad file, the artificial intelligence could simply learn to look for that connection rather than the actual attack behavior. This meant the model was passing the test by recognizing the test's rules, not by understanding the threat.

The team also pointed out a specific flaw in how the data was prepared. In many studies, the artificial intelligence was given a complete picture of the system's history, including events that happened hours or days after the moment an alert should have been raised. This is known as look-ahead bias. For a security system to be useful, it must make a decision based only on what has happened up to that exact second. The review found that very few studies reported whether they had cut off the data at the correct moment. Without this cut-off, the models were essentially being tested on a scenario that does not exist in reality. The researchers also noted that the datasets used for these tests were often small and repetitive, containing only a few weeks of data and a handful of attack scenarios. This limited the ability to see how the models would handle the slow, subtle changes in computer behavior that happen over months or years.

To fix these issues, the authors proposed a new set of standards for how these studies should be conducted. They created a checklist of twenty-one items that researchers must follow to prove their models work in real time. This includes building the data graph only up to the moment of the decision, locking the model's settings before testing it on new data, and reporting how many false alarms the system would generate in a real office environment. They argued that a model that generates too many false alarms is useless, no matter how accurate it is at spotting attacks, because security teams would be overwhelmed. The review concluded that until these new standards are followed, the claims of near-perfect detection in the field of provenance-based security should be viewed with skepticism. The models have shown promise, but the evidence that they can actually protect systems in the wild, against attackers who change their tactics over time, has not yet been demonstrated. The path forward requires a shift from chasing high scores on static tests to proving that these systems can survive the test of time.