🤖 AI

Retrieval-Augmented LLMs for Security Incident Analysis

This paper presents a Retrieval-Augmented Generation (RAG) system that enhances security incident analysis by combining targeted query-based filtering with LLM semantic reasoning to accurately reconstruct attack sequences and identify indicators from diverse log sources, demonstrating that this approach achieves high recall and precision while significantly reducing costs compared to non-RAG baselines.

Original authors: Xavier Cadet, Aditya Vikram Singh, Harsh Mamania, Edward Koh, Alex Fitts, Dirk Van Bruggen, Simona Boboila, Peter Chin, Alina Oprea

Published 2026-03-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xavier Cadet, Aditya Vikram Singh, Harsh Mamania, Edward Koh, Alex Fitts, Dirk Van Bruggen, Simona Boboila, Peter Chin, Alina Oprea

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a massive crime that happened inside a giant, bustling city (your computer network). Every day, millions of people walk through the streets, buy coffee, and talk to neighbors. Most of this is normal life. But somewhere in that sea of millions of transactions, a thief sneaks in, steals a wallet, and runs away.

Your job is to find the thief.

The Problem: Too Much Noise, Too Little Time

In the old days, security analysts were like detectives who had to read every single receipt, security camera tape, and phone call log from the entire city to find the thief.

The Reality: A cyberattack leaves behind millions of digital "receipts" (logs).
The Problem: There are too many receipts. A human can't read them all. They get tired, miss clues, and the thief gets away. Also, the thief is smart; they don't leave a sign that says "I am the thief." They hide their tracks among normal activities.

The Solution: A Smart Detective Assistant (RAG + LLM)

The paper introduces a new system that acts like a super-smart detective assistant. It combines two powerful tools:

The Filter (RAG): A tool that quickly scans the millions of receipts and only keeps the ones that look suspicious.
The Brain (LLM): A highly intelligent AI that reads those few suspicious receipts and tells you the full story of what happened.

Here is how it works, step-by-step, using our city analogy:

1. The "Smart Filter" (Targeted Query-Based Filtering)

Instead of giving the AI the whole library of books (all the logs), the system first asks specific questions to a librarian.

Old Way: "Read every book in the library." (Too slow, too much noise).
New Way: "Show me only the books about 'stolen wallets' and 'strange car movements'."
How it works: The system uses a pre-made list of "suspicious patterns" (like looking for a specific type of fake ID or a car driving in circles). It filters out the millions of normal coffee purchases and keeps only the 100 receipts that look weird.

2. The "Context Builder" (Retrieval-Augmented Generation)

Now the AI has a small pile of suspicious receipts. But a receipt from 9:00 AM might not make sense without the one from 9:05 AM.

The system takes those 100 suspicious receipts and arranges them into a coherent story. It links them together like puzzle pieces.
It uses a technique called RAG (Retrieval-Augmented Generation). Think of this as the AI having a "cheat sheet" or a "search engine" right next to it. When the AI tries to solve the mystery, it doesn't just guess; it looks up the specific evidence it needs from its cheat sheet to back up its answer.

3. The "Detective AI" (The LLM)

Finally, the AI (the Large Language Model) reads this curated, organized story. Because it's not overwhelmed by millions of normal receipts, it can focus on the clues.

It answers questions like: "Who was the thief?" "Where did they go?" "What tools did they use?"
It doesn't just say "A theft happened." It says, "The thief was a user named 'John' on computer 'Desk-5'. They downloaded a fake update, then called a secret server in another country to steal data."

The Results: Speed, Accuracy, and Cost

The researchers tested this system against real cyberattacks (like malware and complex network hacks) and compared different AI "brains" (like Claude, DeepSeek, and GPT).

Accuracy: The best AI models got 100% of the clues right. They found the infected computers and the bad guys' secret servers.
The "No-Filter" Failure: When they tried to feed the AI all the raw logs without the filter, the AI got confused. It could find the victim but missed the bad guys entirely because the "noise" drowned out the signal. It's like trying to hear a whisper in a hurricane.
Cost Savings: They found one AI model (DeepSeek) that was just as smart as the most expensive one (Claude) but cost 15 times less. It's like getting a Ferrari engine for the price of a Honda Civic.
Specialized vs. General: Interestingly, an AI specifically trained on "security" didn't do better than a general smart AI. It turns out, being a good reasoner (connecting the dots) is more important than just knowing security jargon.

Why This Matters

This system changes the game for security teams.

Before: Analysts were drowning in data, missing attacks, and burning out.
Now: The system acts as a force multiplier. It filters the noise, organizes the clues, and tells the human analyst exactly what happened, why it happened, and how to stop it.

In short: This paper describes a system that turns a chaotic mountain of digital noise into a clear, readable story, allowing human experts to catch cybercriminals faster, cheaper, and more accurately than ever before.

1. Problem Statement

Security Operations Centers (SOCs) face a critical bottleneck: the volume and complexity of cyber threats have outpaced the growth of the analyst workforce. Modern Security Information and Event Management (SIEM) systems generate thousands of alerts daily, but analysts can only examine a small fraction, leading to fatigue and missed incidents.

The Challenge: Investigating an incident requires synthesizing evidence from disparate sources (Intrusion Detection System alerts, network traffic, authentication logs). This data is often voluminous (millions of log entries), noisy (mixed with benign activity), and fragmented (no single log entry reveals the full attack chain).
Limitations of Current Approaches:
- Signature-based detection: Cannot generalize to novel attack patterns.
- Direct LLM ingestion: Feeding raw logs directly into Large Language Models (LLMs) is impractical due to context window limits and the "needle in a haystack" problem where attack signals are buried in routine noise.
- Existing ML: Often provides anomaly scores or classifications rather than the narrative reconstruction required for incident response.

2. Methodology

The authors propose a Retrieval-Augmented Generation (RAG) system designed specifically for security incident analysis. The system operates in two modes: Forensic Question Answering (answering specific queries about compromised assets) and Attack Reconstruction (reconstructing the causal narrative of an attack).

System Architecture

The system follows a "Filter before Analysis" design principle, decomposing the task into two main components:

Security Context Extraction (Filtering & Aggregation):
- Instead of passing raw logs to the LLM, the system executes a library of targeted queries against the SIEM (e.g., Suricata, Zeek, Windows Event Logs).
- These queries are mapped to MITRE ATT&CK techniques.
- Aggregation: Queries compress thousands of raw events into semantically rich summaries (e.g., "11 Kerberos authentications from IP X by user Y"). This reduces data volume while preserving cross-event relationships.
- Embedding: These aggregated chunks are embedded using a sentence transformer model (all-mpnet-base-v2) and stored in a vector database (FAISS).
RAG-LLM Analysis (Reasoning & Reporting):
- When a forensic question is posed (e.g., "What is the infected IP?"), the system retrieves the top- $k$ most relevant chunks based on semantic similarity.
- The LLM receives the question, network context (topology), and the retrieved evidence chunks.
- The LLM acts as a cybersecurity analyst, generating evidence-backed answers and reconstructing attack timelines.
- Provider Agnostic: The system supports various LLMs (Cloud APIs like Claude, DeepSeek, GPT; Local models like Llama, Cisco Foundation-Sec).

Evaluation Metrics

The system is evaluated using a structured questionnaire with specific answer types (IP, hostname, user, domain set, IP set).

Matching Functions: Custom matching logic is used (e.g., set intersection for domains/IPs, normalization for hostnames) rather than simple string matching.
Metrics: Precision and Recall are calculated against reference answers derived from manual log analysis and published scenario solutions.

3. Key Contributions

Problem Formulation: Framing security incident analysis as a structured question-answering task over heterogeneous logs, distinguishing between asset identification and attack sequence reconstruction.
Security-Aware RAG Architecture: A novel pipeline that uses targeted IOC (Indicator of Compromise) queries and aggregation to create "semantic chunks." This enables LLMs to perform cross-event semantic reasoning (e.g., correlating a certificate request with a Kerberos ticket to detect privilege escalation) which raw log ingestion fails to achieve.
Comprehensive LLM Provider Evaluation: A rigorous comparison of 8 LLM configurations (6 cloud APIs, 2 local models) across malware and Active Directory attack scenarios, including cost-benefit analysis.
Ablation Studies: Demonstrating the necessity of RAG preprocessing and the impact of context size ( $k$ ) on performance.

4. Key Results

Malware Traffic Analysis (4 Scenarios)

Performance: Claude Sonnet 4 and DeepSeek V3 achieved 100% recall across all four malware scenarios (Fake Authenticator, NetSupport RAT, Koi Stealer, IcedID).
Local Models: Locally deployed models (Llama 3.1:70b, Cisco Foundation-Sec-8B) achieved 90–95% recall, proving viable for data-sensitive environments.
Failure Modes: GPT models (4, 4o, 5-mini, 5.2) struggled with multi-stage attacks (e.g., IcedID), often missing the initial infection vector or C2 servers due to an inability to correlate behavioral aggregations across log sources.
Context Sensitivity: Performance for top models (Claude/DeepSeek) plateaued at $k=7$ retrieved chunks. GPT models showed persistent reasoning errors (e.g., confusing user accounts with machine hostnames) regardless of context size.

Active Directory Attack Analysis

Attack Reconstruction: The system achieved 100% precision and 82% recall in detecting attack steps within 5-minute sliding windows.
Semantic Reasoning: The system successfully identified complex attacks like ESC1 certificate template abuse by correlating distinct events (Certificate Request with SAN:Administrator vs. Certificate Issued with Subject:<user>), a task requiring cross-chunk reasoning that simple pattern matching cannot solve.
Defense Recommendations: The system recommended defensive actions (e.g., revoking certificates, resetting KRBTGT) with 100% coverage of the action types required to stop the attack, though it occasionally suggested operationally infeasible actions (e.g., disabling the Administrator account).

Baseline Comparisons (The "No-RAG" Problem)

No-RAG Baseline: When LLMs were fed raw logs (limited to context windows), they achieved 0% recall for attack infrastructure (C2 servers, malicious domains). They could only identify victim IPs/hostnames present in the initial benign logs.
Suricata-Only Baseline: Relying solely on IDS alerts yielded 40% recall, missing identity attribution (user accounts) which requires cross-source correlation (Kerberos/HTTP logs).
Conclusion: RAG preprocessing is essential for both reducing noise and enabling multi-source correlation.

Cost and Efficiency

Cost-Effectiveness: DeepSeek V3 emerged as the most cost-effective provider, achieving 100% recall at $0.008 per analysis, which is 15x cheaper than Claude Sonnet 4 ($0.12).
Latency: Cloud providers analyzed malware incidents in ~1.2 minutes. Local deployment (Llama) took ~2.5 minutes for malware and ~61 seconds per 5-minute window for AD attacks, demonstrating real-time feasibility.

5. Significance and Conclusion

This paper demonstrates that RAG is not just an optional enhancement but a prerequisite for using LLMs in security incident analysis.

Reasoning > Pretraining: Surprisingly, a security-specialized model (Cisco Foundation-Sec-8B) did not outperform general-purpose models (Llama, Claude, DeepSeek), suggesting that reasoning capability is more critical than domain-specific pretraining for complex forensic tasks.
Scalability: The system bridges the gap between massive log volumes and limited analyst capacity, automating the synthesis of evidence into interpretable reports.
Practical Impact: By combining targeted filtering with semantic retrieval, the system enables accurate, cost-effective, and privacy-preserving (via local deployment) incident response, capable of handling multi-stage attacks that traditional tools miss.

The authors conclude that while the system shows high promise, future work involves validating on a broader range of attack types and larger-scale deployments. The code, query library, and evaluation scripts have been released to support reproducibility.