Detecting Cybersecurity Threats by Integrating Explainable AI with SHAP Interpretability and Strategic Data Sampling

1. Problem Statement

The paper addresses three critical challenges in deploying Machine Learning (ML) for cybersecurity operations, specifically within Security Operations Centers (SOCs):

• The "Black Box" Problem: Advanced models (e.g., Deep Learning, Ensemble methods) often lack transparency, making it difficult for analysts to trust, validate, and act upon detection decisions.
• Data Scale and Efficiency: Modern network datasets (e.g., CIC-IDS2017 with >2.8 million records) are computationally expensive to process, necessitating efficient sampling strategies that do not introduce statistical bias.
• Experimental Integrity: Many existing studies suffer from data leakage (e.g., temporal bias, feature contamination), leading to inflated performance metrics that do not reflect real-world deployment scenarios.

2. Methodology

The authors propose an Integrated Explainable AI (XAI) Framework that unifies strategic sampling, rigorous data hygiene, and model interpretability. The workflow consists of the following key stages:

A. Strategic Data Sampling

• Approach: Instead of synthetic oversampling (like SMOTE) which alters data distribution, the authors employ a Stratified Sub-sampling method.
• Mechanism: A formula is used to select a representative subset (20% of the original data) while strictly preserving the original class distribution ratios.
• Result: Reduced the CIC-IDS2017 dataset from 2.83 million to 470,269 samples, enabling rapid model iteration without sacrificing statistical representativeness.

B. Automated Data Leakage Prevention

• Temporal Splitting: The dataset is partitioned chronologically (Days 1–4 for training/validation, Day 5 for testing) to simulate real-world future attack detection, avoiding random splits that cause temporal leakage.
• Feature Cleaning: An automated process identified and removed 29 "leaky" features (37% of the original 78 features) that exhibited near-perfect correlation with target classes or contained future information.
• Preprocessing Isolation: Feature scaling parameters (mean and standard deviation) were computed only from the training set to prevent information leakage into validation/test sets.

C. Multi-Configuration Algorithm Evaluation

• Validation Strategy: The framework tested three distinct data partitioning schemes to ensure robustness:
  • Config A: 40% Train / 10% Val / 50% Test (Limited data).
  • Config B: 60% Train / 10% Val / 30% Test (Balanced).
  • Config C: 80% Train / 10% Val / 10% Test (Abundant data).
• Algorithms: XGBoost, Random Forest, and Logistic Regression were evaluated across these configurations.
• Feature Selection: Comparative analysis of MRMR (Maximum Relevance Minimum Redundancy) and Chi-squared methods at 30%, 50%, and 70% reduction rates.

D. Integrated Explainable AI (XAI)

• SHAP Analysis: SHAP (SHapley Additive exPlanations) was embedded throughout the pipeline to provide model-agnostic interpretability.
• Output: Generated global feature importance rankings and local decision rationales (explaining why a specific packet was flagged) to aid security analysts.

3. Key Contributions

1. Strategic Sampling Methodology: A novel approach that reduces dataset volume by ~83% while maintaining exact class distributions, facilitating efficient development without synthetic data distortion.
2. Rigorous Leakage Prevention: Systematic removal of 29 leaky features and implementation of strict temporal validation, addressing common pitfalls in cybersecurity ML research.
3. Optimized Feature Selection: Demonstrated that MRMR with 70% reduction (retaining 34 features) not only maintained but slightly improved model performance (F1-Macro +0.05%) while significantly reducing training time.
4. Unified Framework: Successfully integrated sampling, feature optimization, and XAI into a single pipeline, bridging the gap between high-performance algorithms and operational transparency.

4. Results

The framework was evaluated on the CIC-IDS2017 dataset using the optimal configuration (XGBoost with 60-10-30 split and MRMR 70% feature selection):

• Performance Metrics:
  • Accuracy: 99.92%
  • F1-Macro: 99.77%
  • ROC-AUC: 0.99997 (Near-perfect discrimination)
  • Precision/Recall: Balanced performance across both high-frequency (DoS, DDoS) and low-frequency (FTP-Patator, GoldenEye) attack classes.
• Computational Efficiency:
  • Training Time: Reduced from 33.90s (baseline) to 21.47s (36.7% faster) due to feature reduction.
  • Inference Speed: 324,913 predictions per second, demonstrating suitability for real-time deployment.
• Interpretability: SHAP analysis successfully identified key predictive features (e.g., Flow_Bytes/s, Packet_Length_Std, Flow_IAT_Min) and provided class-specific explanations, validating the model's decision logic.

5. Significance and Impact

• Operational Trust: By providing actionable explanations via SHAP, the framework addresses the "black box" barrier, allowing SOC analysts to validate alerts and understand attack patterns.
• Methodological Rigor: The paper sets a new standard for experimental integrity in cybersecurity ML by prioritizing temporal validation and leakage prevention over raw accuracy optimization.
• Scalability: The combination of strategic sampling and feature reduction proves that high-accuracy threat detection is achievable on massive datasets without prohibitive computational costs.
• Deployment Readiness: The framework achieved a 9.1/10 deployment readiness score, indicating it is a robust foundation for real-world Security Operations Centers, though future work is needed to validate performance across diverse environments (IoT, Cloud) and against zero-day attacks.

In conclusion, this research demonstrates that explainability, computational efficiency, and experimental integrity can be simultaneously achieved, offering a trustworthy pathway for deploying AI-driven Intrusion Detection Systems (IDS).