Operationalizing Cyber Attack Prediction: A Gap-Prioritized Framework with Dataset and Model Selection Guidelines

This paper bridges the gap between theoretical research and practical deployment in AI-driven cyber defense by analyzing over 150 datasets and 200 studies to prioritize critical implementation hurdles, introduce a gap-prioritization framework, and provide actionable guidelines for dataset selection and model deployment.

The Gist

Imagine you are trying to build a super-smart security guard for a digital fortress. You want this guard to spot thieves (cyber attackers) before they break in. For years, scientists have been training these guards using old textbooks and practice drills. They claim the guards are 99% perfect at catching thieves.

But here is the problem: The drills are outdated, and the thieves have changed their tactics.

This paper, written by Mr. Aminu Muhammad Auwal, acts as a reality check. It looks at the gap between what scientists say in their labs and what actually works in the real world. The author uses a "gap analysis" to find five major holes in the current system and provides a practical guide to fix them.

Here is the breakdown of the paper's findings using simple analogies:

1. The Five Big Holes (The Gaps)

The author identifies five specific reasons why these "perfect" AI guards fail in real life:

• The "Old Textbook" Problem (Temporal Obsolescence):
  Imagine training a firefighter to put out fires using a manual from 1998. Today's fires are caused by lithium batteries and smart home devices, but the firefighter is still looking for wood and kerosene.

  • The Paper's Claim: Many AI models are trained on datasets (data collections) that are 8 to 15 years old. They don't know about modern threats like AI-powered phishing or deepfakes. It's like trying to defend a modern city with 1990s police tactics.

• The "One-Tool" Problem (Narrow Attack Scope):
  Imagine a security guard who only knows how to stop people climbing over a fence. If a thief walks through the front door or uses a key, the guard doesn't react.

  • The Paper's Claim: Most datasets only teach the AI about a few types of attacks (like 3 or 4). Real life has dozens of different ways to attack. If the AI hasn't seen a specific type of attack in its training, it won't catch it.

• The "Black Box" Problem (Interpretability):
  Imagine a security guard who screams "THIEF!" but refuses to tell you why or where the thief is. You can't trust them if you don't understand their logic.

  • The Paper's Claim: The most accurate AI models are "black boxes." They give an answer but can't explain how they got there. Human security teams need to know why an alert was triggered to take action, but the AI won't tell them.

• The "Trickster" Problem (Adversarial Robustness):
  Imagine a guard who is great at spotting a thief in a black hoodie. But if the thief puts on a bright yellow hat, the guard ignores them. The thief just needs to change one small thing to fool the guard.

  • The Paper's Claim: Hackers can make tiny, invisible changes to their attacks to trick the AI. The current research doesn't test enough to see if the AI can handle these tricks.

• The "Privacy" Problem (Ethics):
  Imagine a guard who watches everyone's private conversations to find bad guys. Even if they catch the bad guys, they might be breaking the law or making people feel unsafe.

  • The Paper's Claim: AI systems often need to look at private data to work, but there aren't enough rules or guidelines on how to do this without violating privacy or fairness.

2. The Solution: A Prioritization Framework

The author doesn't just list problems; they give you a "To-Do List" based on what is easiest and most effective to fix first. They scored the problems based on Impact (how bad is it?), Cost (how much money/time?), and Time (how fast can we fix it?).

• The "Quick Win" (Highest Priority): Fix the Black Box problem.
  • Why? It's relatively cheap and fast to add "Explainable AI" (XAI). This is like giving the guard a walkie-talkie so they can say, "I see a thief because they are running and holding a bag." This builds trust and helps humans make decisions immediately.
• The "Big Project" (Critical but Hard): Fix the Old Textbook problem.
  • Why? This is the most dangerous gap (using old data), but it's expensive and slow to fix because you need to collect brand-new data. It's essential for long-term safety but not a quick fix.
• The "Middle Ground": Fixing the "One-Tool" problem and the "Trickster" problem requires more resources and time.

3. The Practical Roadmap (How to Build Your Guard)

The paper gives a step-by-step guide for organizations of different sizes:

• For Small Organizations (Limited Budget):

  • Don't try to build a super-complex AI from scratch.
  • Do use "Random Forest" (a specific type of AI that is accurate, cheap to run, and easy to understand).
  • Do use public datasets that are newer (like CICIDS2017) instead of the old ones.
  • Do add "Explainable AI" tools immediately so you know why the system is alerting you.

• For Large Organizations (Big Budget):

  • You can afford to build your own private datasets (so you aren't using old public ones).
  • You can use complex Deep Learning models (like CNNs or LSTMs) for better pattern recognition.
  • You should test your system against "tricksters" (adversarial testing) to make sure it can't be fooled.

Summary

The paper argues that we have been celebrating AI security models that look great on paper but fail in the real world because they are trained on old data, can't explain themselves, and are easily tricked.

The author's main message is: Stop trying to build the most complex AI immediately. Instead, start by making your AI explainable (so humans trust it), use newer data, and follow a step-by-step plan based on how much money and time you have. This bridges the gap between "science fiction" and "real-world security."

Technical Summary

Technical Summary: Operationalizing Cyber Attack Prediction

Problem Statement
Despite significant advancements in Artificial Intelligence (AI) and Machine Learning (ML) for cyber attack prediction, a critical disconnect persists between theoretical research capabilities and practical deployment in operational environments. While academic literature reports high detection accuracies (e.g., Random Forest achieving 99.92% on UKM-IDS20), security practitioners struggle to implement these systems effectively. This "research-practice divide" is driven by five primary limitations: the use of temporally obsolete datasets that fail to represent contemporary threats; narrow attack scope coverage that limits model generalization; the "black box" nature of deep learning models which hinders real-time interpretability; insufficient adversarial robustness testing; and a lack of practical frameworks for addressing privacy and ethical concerns.

Methodology
This study conducts a systematic gap analysis building upon the comprehensive survey by Ankalaki et al. (2025), which reviewed over 200 research studies and 150+ benchmark datasets. The methodology involves:

1. Gap Identification: Analyzing the survey findings to categorize five critical barriers to real-world implementation.
2. Gap Prioritization Framework: Developing a multi-dimensional scoring system to evaluate each gap based on three axes: Impact on detection effectiveness ($I$), Implementation cost ($C$), and Time to address ($T$). A priority score is calculated using the formula: $Priority = I \times (11 - \frac{C+T}{2})$.
3. Dataset Quality Assessment Framework (DQAF): Creating a decision-support tool to classify 45 benchmark datasets into three categories—Production-Ready, Research-Only, and Unusable—based on temporal currency, attack scope, traffic realism, and availability.
4. Implementation Roadmap: Synthesizing these findings into actionable guidelines for dataset selection, model selection, Explainable AI (XAI) integration, and ethical deployment, tailored to organizational resource constraints.

Key Contributions
The paper makes four primary contributions to the field of AI-driven cybersecurity:

1. Critical Gap Analysis: It identifies and quantifies five specific gaps hindering deployment: temporal dataset obsolescence, narrow attack scope, real-time interpretability challenges, inadequate adversarial robustness, and unaddressed ethical considerations.
2. Gap Prioritization Framework: It introduces a quantitative matrix that helps organizations allocate resources by ranking gaps. The analysis reveals that while dataset obsolescence and adversarial robustness have high impact, Real-Time Interpretability offers the highest overall priority score (56.0) due to its high impact combined with low cost and short implementation time.
3. Dataset Quality Assessment Framework: It classifies 45 datasets, identifying only four as "Production-Ready" (Edge-IIoTset, CICIDS2017, Bot-IoT, and UNSW-NB15). It explicitly categorizes widely used legacy datasets like NSL-KDD (2009) and DARPA 1998 as "Research-Only" or "Unusable" for production due to their 16–27-year age gaps.
4. Practical Implementation Roadmap: It provides a phased, resource-aware guide for practitioners. This includes decision trees for dataset selection, comparative performance tables for ML/DL models (highlighting Random Forest as optimal for cost-performance balance), a three-phase XAI integration strategy, and checklists for ethical deployment.

Results
The application of the prioritization framework yields specific strategic insights:

• XAI as a High-Value Lever: Integrating Explainable AI (specifically SHAP and LIME) is identified as the most cost-effective immediate improvement, addressing the trust and accountability issues of "black box" models without requiring massive resource investment.
• Dataset Obsolescence: The analysis confirms that models trained on datasets older than 8–15 years (e.g., NSL-KDD, DARPA 1998) possess a fundamental intelligence deficit, rendering them ineffective against modern threats like AI-powered phishing and LLM-based malware.
• Model Selection: Random Forest is highlighted as the most suitable baseline for resource-constrained organizations, offering high accuracy (~99.2% average) and inherent interpretability. Deep Learning models (CNNs, LSTMs) are noted for high accuracy but require significant computational resources and external XAI integration to be operationally viable.
• Strategic Tiering: The framework categorizes actions into Tier 1 (Critical: Interpretability and Dataset Currency) and Tier 2 (High/Medium: Attack Scope, Robustness, Ethics), providing a clear path for organizations of varying sizes to prioritize improvements.

Significance
The paper claims significance by translating comprehensive survey findings into practical decision-support tools, directly addressing the need for production-oriented guidance in AI-driven cyber defense. By shifting the focus from purely academic accuracy metrics to operational viability (considering cost, time, and interpretability), the study enables security practitioners to navigate the complex landscape of cybersecurity AI research. It argues that effective cyber defense requires not just accurate prediction, but systems that are interpretable, robust, ethical, and trained on current data—bridging the gap between theoretical potential and operational reality.