Entity Embeddings and Ensemble Learning for Multi-Attribute Cyber-Attack Forecasting: A Benchmark and Evaluation Protocol for Merged Incident Repositories

In the high-stakes world of cybersecurity, defenders have long operated in a reactive mode, waiting for an alarm to sound before they can act. By the time a breach is detected, the damage is often already done. To change this dynamic, researchers have turned to forecasting: the attempt to predict what kind of attacks will happen, where they will strike, and who will be targeted before they occur. This field relies on vast collections of historical data, known as incident repositories, which record the details of past cyberattacks. However, these records are often scattered across different databases, each using its own language and format, and many are missing crucial details. To build a useful picture of the future, scientists must merge these separate archives into a single, complete story, filling in the missing gaps with educated guesses derived from the data they do have. The challenge lies in ensuring that these guesses do not accidentally trick the very models designed to learn from them, creating a false sense of security where none exists.

A team of researchers at the International Islamic University in Islamabad, Seoul National University, and the University of Prince Mugrin set out to test the reliability of this common practice. They combined two massive databases of cyber incidents: the VERIS Community Database, which offers deep, structured details on attacks but stops recording in 2017, and Hackmageddon, which provides a continuous stream of recent incidents but with less detailed information. By harmonizing these two sources, they created a unified record of over 20,000 incidents spanning more than fifty years. To make the data usable, they had to fill in missing fields for each incident, such as the specific motive of the attacker or the type of asset targeted. They did this by training a computer model to predict the missing details based on the information available in the other database. This process produced a complete dataset, but it also introduced a subtle trap. Because the missing labels were generated by a computer model using the same features that the forecasting models would later use to make predictions, the forecasting models were essentially being tested on answers they had already been trained to recognize.

When the researchers evaluated their forecasting models on this merged dataset, the results initially looked promising. The models appeared to predict the type of future attack with an accuracy that seemed to outperform standard benchmarks. However, the team suspected that these high scores were an illusion caused by the way the data was constructed. To investigate, they separated the test data into two groups: records where the attack type was originally observed and recorded by humans, and records where the attack type had been reconstructed by the computer model during the merging process. The difference was stark. On the records with human-observed labels, the model's performance dropped significantly, revealing that the high scores on the full dataset were largely driven by the model simply memorizing the patterns of the computer that had filled in the missing data. In fact, the gap between the two groups was so large that the overall average performance was inflated by nearly forty-four percentage points, a distortion that would have remained hidden if the researchers had only looked at the combined numbers.

To confirm that this inflation was caused specifically by the act of filling in the missing data, and not by other differences between the two time periods, the researchers performed a controlled experiment. They took a single time period where all the data was originally observed, then deliberately hid a portion of the labels and filled them in using the same computer model they had used for the main dataset. When they trained their forecasting model on this mixed data, the performance on the hidden, reconstructed labels was once again much higher than on the observed ones. This proved that the act of imputation itself creates a circular loop where the model learns to mimic the filler rather than the real-world phenomenon. The study also revealed a counter-intuitive finding: the worse the computer model was at filling in the missing data, the more the forecasting model's performance was artificially inflated. A poor imputer creates a unique, idiosyncratic pattern in the missing data that is easy for a downstream model to learn, whereas a highly accurate imputer blends in so well that it does not create such a distinct, learnable signal.

Beyond the issue of data leakage, the researchers explored how to best represent the information within these records. Cyber incident data is filled with high-cardinality categories, such as the names of hundreds of different countries or thousands of specific organizations. Traditional methods often treat these as simple numbers, which can confuse the models. The team tested a technique called entity embeddings, which learns a unique, dense mathematical representation for each category, allowing the model to understand the relationships between them. This approach dramatically improved the model's ability to predict attack types, boosting its performance by nearly nineteen percentage points compared to the traditional method. This finding suggests that previous studies claiming deep learning models perform poorly on this type of data may have been limited by how the data was presented to them, rather than by the models themselves.

The study also examined the difficulty of predicting the specific country targeted by an attack. While the models could achieve a high overall accuracy by simply guessing the most common countries, they struggled immensely to correctly identify the less frequent targets. The performance on this task was so low that it barely exceeded the baseline of always guessing the majority class. The researchers determined that this was not due to a lack of data or an imbalance in the number of attacks, but rather because there simply is not enough specific information in the available records to distinguish between the hundreds of different countries. The features present in the data do not carry the necessary geographic signal to make these fine-grained predictions reliable.

Finally, the team tested their models under a strictly chronological evaluation, training on data from the past and testing on data from the future, which is the only way to truly assess a forecasting system. Under these realistic conditions, the performance dropped further, and the deep learning models, which had previously trailed behind simpler tree-based models, finally surpassed them. This reversal suggests that the deep models, aided by the better representation of the data, were better at generalizing to new, unseen time periods, while the simpler models were merely memorizing patterns specific to the training window. The researchers concluded that while forecasting the general type of attack is possible with modest success, predicting the specific country or the exact pattern of an attack remains out of reach with current data and methods. Their primary contribution was not a new prediction algorithm, but a rigorous protocol for evaluating merged datasets that exposes these hidden flaws, ensuring that future research in cybersecurity and other fields does not mistake the artifacts of data construction for genuine predictive power.