💻 computer science

Evaluating Retrieval-Augmented Generation for Explainable Malware Analysis

This paper empirically demonstrates that Retrieval-Augmented Generation (RAG) often degrades the quality of malware explanations by introducing noise and irrelevant context, suggesting that malware analysis is better treated as a signal-extraction task rather than a knowledge-retrieval problem when structured evidence is already available.

Original authors: Jayson Ng, Amin Milani Fard

Published 2026-05-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jayson Ng, Amin Milani Fard

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery: Why is this computer file acting like a criminal?

In the world of cybersecurity, analysts have a massive database called VirusTotal. Think of this database as a super-detailed police report. It lists exactly what the file did: "It opened a door," "It copied a file," "It called a suspicious phone number." It's all there, in plain, structured facts.

Recently, security experts started using AI detectives (Large Language Models, or LLMs) to read these police reports and write a clear story explaining why the file is bad.

The Big Question: Do We Need a "Library" for the AI?

There is a popular idea in AI called RAG (Retrieval-Augmented Generation). The logic goes like this: "If our AI detective is writing a story, let's give it a library of extra books to read first. Maybe it needs to look up what 'opening a door' usually means in other crime stories to write a better explanation."

The authors of this paper decided to test if this "library" actually helps when the police report (VirusTotal) is already perfect and complete.

The Experiment: The Detective with and without the Library

The researchers set up a test with two groups of AI detectives:

Group A: Read the VirusTotal report and wrote an explanation.
Group B: Read the VirusTotal report, plus the AI went to the "library," grabbed some extra books about malware, and tried to mix those stories into the explanation.

They used a scoring system (called BERTScore) to see how well the AI's story matched the "gold standard" of a perfect explanation.

The Surprising Result: The Library Made Things Worse

The paper found that adding the extra library books actually made the AI's explanations worse.

Here is the analogy:
Imagine you are trying to explain a car crash to a judge. You have the official police report with the speed, the skid marks, and the weather.

Without the library: The AI looks at the report and says, "The car was speeding and hit the tree." (Clear, accurate).
With the library: The AI goes to the library, finds a book about "How trees grow," and another book about "The history of speed limits." It tries to mix these into the story. Suddenly, the explanation becomes: "The car was speeding, and while trees usually grow slowly, this one was hit by a fast car, which reminds us of the 1990s speed limit debates..."

The extra information didn't help; it distracted the AI. It added "noise" and made the story confusing or generic.

Why Did This Happen?

The authors explain that analyzing malware is like finding a needle in a haystack, not like writing an essay from scratch.

The VirusTotal report already has all the "needles" (the specific bad behaviors).
The AI's job is just to extract those needles and tell you what they mean.
When you force the AI to read extra books (RAG), it starts looking at "straw" (irrelevant or weakly related info) instead of the needles. It gets confused by stories that sound related but aren't actually about this specific file.

The Takeaway for Security Teams

The paper concludes that for this specific job (explaining malware based on a complete report), less is more.

Don't overcomplicate it: If you already have the full facts, don't force the AI to look up extra information. It just makes the AI hallucinate or get distracted.
Signal vs. Noise: The job is to pull out the clear signals from the data, not to synthesize new knowledge from a library.
Better Tools: Instead of using a "library" approach, security tools should focus on helping the AI read the existing report more carefully, perhaps by organizing the facts better or ignoring irrelevant distractions.

In short: When the evidence is already right in front of you, sending the AI to the library just makes it forget what it was supposed to be looking at.

Technical Summary: Evaluating Retrieval-Augmented Generation for Explainable Malware Analysis

Problem Statement

While Large Language Models (LLMs) are increasingly utilized to translate low-level malware indicators (e.g., API calls, registry edits) into natural language explanations, the efficacy of Retrieval-Augmented Generation (RAG) in this specific domain remains unverified. The prevailing assumption is that injecting external security knowledge via RAG improves explanation quality. However, malware analysis often relies on structured evidence (such as VirusTotal reports) that already encapsulates sufficient contextual data. The authors posit that indiscriminate retrieval in such scenarios may introduce "narrative noise," distracting context, or generic write-ups that degrade the model's ability to extract salient signals, potentially leading to misleading explanations in security-critical pipelines.

Methodology

The study conducts an empirical evaluation of LLM explainability with and without RAG, using structured VirusTotal reports as the primary input.

Dataset and Architecture: The dataset comprises 1,702 VirusTotal reports from the MalGPT corpus, containing structured indicators like API calls and registry operations. The RAG pipeline utilizes LlamaIndex for orchestration, ChromaDB for vector storage, and OpenRouter as the LLM gateway. Embeddings are generated using all-MiniLM-L6-v2 and OpenAI's text-embedding-3-large. Documents are chunked into 1,024 tokens with a 100-token overlap, and retrieval employs top- $k$ similarity search with a 0.5 threshold.
Knowledge Base: The retrieval source includes the MITRE ATT&CK framework, research guidance (e.g., MalGPT, CISA/MITRE mappings), VirusTotal technical documentation, and malware intelligence datasets (taxonomies, CVEs).
System Prompt: A structured 10-section prompt enforces consistency, requiring outputs to include verdicts, non-technical summaries, evidence citations, malware family mapping, IoC grouping, behavioral summaries, confidence scores, and actionable recommendations.
Evaluation Protocol: An ablation study compares various LLMs (GPT OSS 21B, DeepSeek-R1 671B, and GPT-5.1) under identical prompts, testing them with VirusTotal JSON alone versus JSON augmented with retrieved context. No fine-tuning is applied to isolate retrieval effects.
Metrics: Explanation quality is measured using BERTScore, which evaluates semantic similarity against a GPT-5.1 baseline (without RAG). This metric is chosen for its robustness to paraphrasing and sensitivity to semantic drift caused by loosely related retrieved passages.

Key Results

The experiments reveal that RAG frequently degrades explanation quality rather than enhancing it when structured evidence is already present:

Performance Degradation: Across multiple models, the addition of retrieved context generally lowered BERTScores. For instance, DeepSeek-R1 achieved its highest score (0.8617) without RAG, while performance dropped to 0.8583 and 0.8582 with different embedding models. Similarly, GPT OSS 21B saw a significant drop from 0.8407 (no RAG) to 0.8184 with the all-MiniLM-L6-v2 retriever.
Embedding Sensitivity: While text-embedding-3-large offered marginal improvements for GPT OSS 21B, it provided no benefit to DeepSeek-R1.
Cause of Degradation: The authors attribute the decline to the retrieval of semantically similar but low-utility passages (e.g., generic malware descriptions) that act as "hard negatives." These distractors shift model attention toward plausible but irrelevant narratives, diluting the specific signals found in the structured VirusTotal reports.
Model Behavior: Stronger models (like DeepSeek-R1) appear to treat the additional context as redundant or mildly contradictory, further supporting the hypothesis that the task is one of signal extraction rather than knowledge completion.

Key Contributions

Empirical Challenge to RAG Assumptions: The work provides evidence that the default application of RAG to malware explanation is counterproductive when high-quality structured evidence (VirusTotal reports) is available.
Task Reframing: The authors argue that malware explanation is primarily a signal-extraction task, not a knowledge-retrieval problem. Consequently, adding external context can introduce cognitive load and error risks without adding value.
Design Recommendations: The paper outlines specific strategies to mitigate retrieval-induced noise in security workflows:
- Skipping RAG when structured reports are sufficient (using context-sufficiency predictors).
- Applying domain-tuned re-rankers with strict $k$ limits to avoid hard negatives.
- Replacing fixed-window chunking with behavior- or IoC-centric chunks enriched with metadata.
- Exploring alternatives like Cache-Augmented Generation (CAG) and Context Engineering.

Significance and Claims

The paper claims that in security-critical pipelines, the risk of automation bias increases when RAG-based explanations drift from the underlying evidence. The authors assert that analysts benefit more from consistency and traceability than from expanded narratives. By demonstrating that RAG can degrade performance in domains driven by structured reasoning, the work advocates for a shift in tooling design: moving away from knowledge synthesis via retrieval and toward structured signal extraction. The findings suggest that for tasks where the source evidence is already comprehensive, the "noise" introduced by retrieval mechanisms outweighs the potential benefits of external knowledge injection.