XAI and Statistical Analysis for Reliable Intrusion Detection in the UAVIDS-2025 Dataset: From Tree to Hybrid and Tabular DNN Ensembles

Technical Summary: XAI and Statistical Analysis for Reliable Intrusion Detection in the UAVIDS-2025 Dataset

Problem Statement
The paper addresses the challenge of detecting intrusions in Unmanned Aerial Vehicle (UAV) networks, specifically within the context of the emerging UAVIDS-2025 dataset. While Explainable Artificial Intelligence (XAI) has gained traction for interpreting complex machine learning models, a critical gap remains in understanding the "Density Support Intersection" between specific attack types. The authors identify a significant difficulty in distinguishing between "Blackhole" and "Wormhole" attacks, where these two classes exhibit high overlap in their feature distributions, leading to misclassifications even in high-performing models. The goal is to develop robust, reliable, and explainable intrusion detection systems (IDS) that not only achieve high accuracy but also provide statistical evidence for why specific misclassifications occur.

Methodology
The research follows a structured pipeline involving data pre-processing, model training, explainability analysis, and statistical validation:

1. Data Pre-processing:

   • Dataset: Utilization of the UAVIDS-2025 dataset (~120,000 observations, 22 features) containing Normal traffic and four attack types: Sybil, Blackhole, Wormhole, and Flooding.
   • Cleaning: Removal of duplicates, the redundant "Protocol" feature (all UDP), and the "FlowID" indexing variable.
   • Encoding: Label encoding for the target, dummy encoding for ports, and frequency encoding for IP addresses.
   • Feature Engineering: Application of power and reciprocal transformations. Outliers were handled using the Interquartile Range (IQR) and Box-Plots, leading to the selection of a Robust Scaler.
   • Feature Selection: Recursive Feature Elimination (RFE) with Random Forest selected features with >2.5% contribution. To prevent multicollinearity affecting SHAP analysis, features with Pearson/Kendall correlation >0.7 were removed, resulting in a final set of 8 features (e.g., LostPackets, RxBytes, PacketDropRate, MeanDelay/s).

2. Model Training and Evaluation:

   • Algorithms: A comprehensive comparison of tree ensembles (XGBoost, LightGBM, Random Forest, etc.), deep neural networks (MLP, RealMLP), hybrid stacking models (combining trees, linear, and Bayesian models), and ensemble neural networks (from AutoGluon and PyTabKit).
   • Validation: Stratified 10-fold cross-validation with hyperparameter tuning via Grid-search. Probability calibration was applied, with a 4% stability threshold for metrics.
   • Metrics: Performance was evaluated using Precision, Recall, F1-score, and ROC-AUC.

3. Explainability (XAI):

   • SHAP Analysis: The top-performing model (XGBoost) was analyzed using Shapley Additive explanations (SHAP) to determine global and local feature importances. This included examining how specific features contribute to predictions for individual observations, particularly in cases of misclassification.
   • Distribution Analysis: Visual inspection of data shapes using Box-Plots and Violin Plots to identify outliers, skewness, and multi-modal distributions. Kernel Density Estimation (KDE) with bandwidth optimization was used to visualize probability densities.

4. Statistical Validation:

   • Westfall-Young Permutation Test: To rigorously investigate the "Density Support Intersection" between Blackhole and Wormhole attacks, the authors applied a non-parametric Westfall-Young test with $B=1000$ permutations.
   • Test Statistic: The Jensen-Shannon Distance (JSD) was used as the metric to compare density estimations. The test calculated Family-Wise Error Rate (FWER) adjusted p-values to handle multiple comparisons without relying on Bonferroni corrections.

Key Results

• Model Performance: Tree-based ensembles demonstrated superior performance. XGBoost achieved the highest results with a Test Precision of 95.17%, Recall of 95.04%, and F1-score of 95.04%. Deep learning models like RealMLP also performed well (~94% F1), while baseline models like Logistic Regression struggled (<50%).
• Misclassification Patterns: The confusion matrices revealed that XGBoost occasionally misclassified Wormhole attacks as Normal traffic and confused Blackhole attacks with Wormhole attacks (and vice versa).
• Feature Importance: Global SHAP analysis identified PacketDropRate, RxByteRate/s, RxBytes, and DstPort654 as the most impactful features. However, local analysis showed that for misclassified instances, features like MeanDelay/s, AverageHopCount, and MeanJitter/s played a disproportionately large role.
• Statistical Findings: The Westfall-Young test confirmed that for most features (e.g., LostPackets, MeanDelay/s), the distributions of Blackhole and Wormhole attacks are statistically different ($p < 0.001$). However, the analysis of KDEs revealed a Density Support Intersection: despite statistical differences in shape, the distributions overlap significantly in specific regions (e.g., the overlap zone [0.25, 1.4] for PacketDropRate). This overlap mimics the behavior of one attack type within the other, causing the model's hesitation and subsequent errors.

Significance and Claims
The paper claims to provide a robust framework for UAV intrusion detection that goes beyond simple accuracy metrics. Its primary contributions are:

1. Explainable Reliability: By combining SHAP with statistical distribution analysis, the authors clarify that misclassifications are not due to weak training or poor pre-processing, but are inherent to the dataset's nature—specifically the Density Support Intersection between Blackhole and Wormhole attacks.
2. Statistical Rigor: The application of the Westfall-Young permutation test with JSD provides probabilistic evidence of the "masked" nature of these attacks, offering a method to detect support intersections that standard metrics might miss.
3. Practical Insight: The study demonstrates that even with an "outstanding" performing model like XGBoost, the challenge of overlapping densities in critical features (like PacketDropRate and MeanDelay/s) remains a fundamental barrier to perfect classification in the UAVIDS-2025 dataset.

The authors conclude that their work offers a solid view of the attacks by combining best-practice pre-processing, state-of-the-art ensemble models, and deep statistical analysis, providing a foundation for future work involving cross-dataset validation and advanced architectures like Deep-CNNs and RNNs.