From Detection to Response: A Deep Learning and Retrieval-Augmented Generation Framework for Network Intrusion Mitigation

Technical Summary: From Detection to Response

Problem Statement
While machine learning-based Intrusion Detection Systems (IDS) have achieved high accuracy in classifying network attacks, they frequently fail to address the critical operational need of security analysts: determining the appropriate response. Current deep learning models often function as "black boxes," providing prediction labels without explaining the decision logic. Furthermore, existing Explainable AI (XAI) techniques, such as SHAP and LIME, identify influential features but do not offer actionable mitigation guidance. Additionally, authoritative cybersecurity knowledge sources (e.g., NIST guidelines, MITRE ATT&CK) are too extensive to consult manually during real-time incidents. This paper addresses the gap between high-accuracy threat detection and the generation of structured, evidence-based, and actionable response strategies.

Methodology
The authors propose a unified, end-to-end framework that integrates deep learning-based intrusion detection with a Retrieval-Augmented Generation (RAG) pipeline. The system operates through five modular stages:

1. Data Acquisition and Preprocessing: The framework utilizes two benchmark datasets, CICIDS2018 and UNSW-NB15. To enable cross-dataset evaluation, a unified 66-dimensional feature space is constructed. Categorical features are one-hot encoded, and zero-padding is applied to bridge dimensional gaps. A feature interpretation dictionary maps technical feature names to human-readable descriptions and security implications.
2. Confidence-Calibrated DNN Classification: Instead of a single multi-class model, the system employs an ensemble of three independently trained binary Deep Neural Networks (DNNs) using a one-vs-rest strategy (Benign-vs-Rest, DoS-vs-Rest, DDoS-vs-Rest).
   • Architecture: The models utilize fully connected layers with ReLU activation. The CICIDS2018 model uses three hidden layers, while the UNSW-NB15 model uses four, incorporating batch normalization and dropout regularization.
   • Class Imbalance Handling: To address severe class imbalance (e.g., DoS traffic comprising only 0.64% of UNSW-NB15), the system employs class-weighted binary cross-entropy loss, random oversampling, controlled undersampling, and F1-score-based model selection.
   • Output: The ensemble produces a predicted class and a scalar confidence score, which is mapped to a discrete "SOC Triage Tier" (Very High, High, Medium, Low) to guide analyst prioritization.
3. Explanation-Aware Prompt Construction: Upon classification, the system extracts the top-5 most influential features using gradient-based attribution. These features are mapped to natural language descriptions via the interpretation dictionary. A structured prompt is then assembled containing:
   • Detection context (attack class, confidence, tier).
   • Network flow metadata (IPs, ports, timestamps).
   • Anomalous indicators (top-5 features and their interpretations).
   • A system role instruction defining the LLM as a senior cybersecurity analyst with explicit citation requirements.
4. Hybrid RAG Retrieval: The prompt is augmented by retrieving relevant guidance from a knowledge base constructed from 14 authoritative documents (including NIST Special Publications and MITRE ATT&CK). The retrieval pipeline uses a hybrid approach:
   • Lexical Search: BM25 ranking.
   • Semantic Search: Dense vector retrieval using FAISS and sentence transformers.
   • Query Expansion: A cybersecurity-specific synonym dictionary expands terms (e.g., "DoS" to "resource exhaustion").
   • Reranking: A cross-encoder reranks the top candidates to select the final top-5 chunks for the prompt.
5. LLM-Based Report Generation: A locally deployed LLaMA 3:8B model (via Ollama) generates a structured incident response report. The report includes sections on Rationale, Key Indicators, Confidence Assessment, Threat Assessment, and Recommendations, all grounded in the retrieved citations.

Key Contributions

1. Structured Prompting Strategy: A novel method that binds LLM reasoning to concrete detection evidence (predicted class, confidence, and top-5 anomalous features) rather than generic queries.
2. Hybrid Retrieval Pipeline: A retrieval system combining BM25 keyword search, FAISS-based dense vector search, and cross-encoder reranking to extract relevant mitigation strategies from authoritative sources.
3. End-to-End Mitigation Framework: A system that not only detects attacks but generates structured, citation-grounded mitigation reports, bridging the gap between detection and response.
4. Rigorous Evaluation: An evaluation framework comparing RAG-enhanced outputs against expert-written ground truth and vanilla LLM baselines using BERTScore, ROUGE, and BLEU metrics.

Results

• Detection Performance: The ensemble DNN achieved 99.84% accuracy on the CICIDS2018 dataset and 95.30% on the UNSW-NB15 dataset. Notably, DoS recall on the highly imbalanced UNSW-NB15 dataset improved from 0.30% (baseline) to 97.6% after applying balancing interventions.
• Retrieval Performance: The hybrid retrieval pipeline achieved a 97% success rate with a mean reciprocal rank (MRR) of 0.87 and an average latency of 1.8 seconds.
• Report Quality: RAG-enhanced reports significantly outperformed vanilla LLM outputs across all metrics. Compared to the baseline, the RAG system showed a 4.4% improvement in BERTScore F1, a 32.5% increase in ROUGE-1, a 126.4% increase in ROUGE-2, and a 244.1% increase in BLEU. The generated reports consistently included citations to NIST and MITRE sources (93.3% and 86.7% citation rates, respectively).

Significance
The paper claims that its framework successfully addresses the persistent limitation in IDS research where detection is decoupled from actionable response. By integrating deep learning detection with a RAG pipeline grounded in trusted cybersecurity knowledge, the system produces mitigation reports that are not only accurate but also explainable, context-aware, and operationally useful for Security Operations Center (SOC) analysts. The authors emphasize that this approach reduces the time between attack detection and effective response, thereby mitigating the risk of prolonged system disruption. The use of a locally deployed LLM also addresses privacy and regulatory concerns associated with routing sensitive network traffic data to external APIs.