💻 computer science

AsmRAG: LLM-Driven Malware Detection by Retrieving Functionally Similar Assembly Code

AsmRAG is a malware detection framework that utilizes Assembly-Level Retrieval-Augmented Generation and a density-weighted anchor selection mechanism to provide highly accurate, interpretable, and obfuscation-resistant forensic analysis by retrieving functionally similar code snippets.

Original authors: ElMouatez Billah Karbab

Published 2026-04-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: ElMouatez Billah Karbab

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a security guard at a massive, high-tech airport.

Currently, most airport security systems work like a "Vibe Check." They look at a passenger’s suitcase, scan the overall weight, the texture of the fabric, and the general "energy" of the person. If the suitcase looks 99% like a typical traveler's bag, the system says, "This person is 99% likely to be a good traveler."

The problem? A clever smuggler can change the color of the bag, add some heavy books to change the weight, or wear a different hat to change their "vibe." The security guard knows something is wrong, but when asked why, they can only shrug and say, "I just have a feeling." This is how current AI malware detectors work—they give a "probability score" but can't tell you exactly what the bad guy is doing.

AsmRAG is different. It’s not a "Vibe Check"; it’s a "DNA Match."

The Core Idea: From "Vibes" to "Blueprints"

Instead of looking at the whole "suitcase" (the entire malware file), AsmRAG breaks the suitcase down into tiny, individual components—like looking at every single screw, wire, and circuit board inside a device. In computer terms, it breaks the software down into its smallest functional instructions, called Assembly Code.

Here is how the AsmRAG process works, using a Master Detective analogy:

1. The Great Library (The Knowledge Base)

Imagine a detective has a massive library containing the "blueprints" of every known criminal tool ever made—the exact way a specific lockpick is shaped or how a specific bomb detonator is wired. This is the Knowledge Base.

2. The Semantic Translator (The LLM)

Criminals are sneaky. They might use a different brand of screwdriver or a slightly different shaped wire to do the exact same job. This is called "obfuscation."
AsmRAG uses a specialized AI (a Large Language Model) that acts like a Master Translator. It doesn't care if the wire is red or blue; it understands that "this wire is being used to complete a circuit." It looks past the "surface" look and understands the intent.

3. The "Anchor" Search (Retrieval)

When a suspicious new file arrives, AsmRAG doesn't just guess. It takes the "blueprints" of the suspicious parts and runs into the library to find a match.
It looks for the "Anchor Function"—the one specific piece of code that is the "smoking gun." It’s like finding a specific, rare type of fingerprint on a stolen diamond. Even if the thief wore gloves, the way they handled the diamond leaves a unique "semantic" trace.

4. The Forensic Report (The Explanation)

This is the most important part. Instead of a security guard saying, "I think this guy is bad," AsmRAG hands the human analyst a written report that says:

"This file is malicious. I am 96% sure because, on page 4, it contains a specific sequence of instructions that is identical to the 'encryption engine' used by the 'Ramnit' gang of hackers. Even though they changed the names of the parts, the way they move data is a perfect match."

Why does this matter?

It’s Hard to Fool: Because AsmRAG looks at the logic (the "why") rather than the appearance (the "what"), hackers can't easily trick it by just adding "junk" code or changing the file's look.
It Explains Itself: It turns a "Black Box" (an AI that gives answers without reasons) into a "Glass Box" (an AI that shows its work). This helps human experts make much faster, more confident decisions.
It Learns on the Fly: Every time a human expert confirms a new type of threat, AsmRAG adds that new "blueprint" to its library instantly. It gets smarter with every single catch.

In short: AsmRAG stops guessing if a file "feels" like malware and starts proving it by finding the digital fingerprints of known criminals.

Technical Summary: AsmRAG (LLM-Driven Malware Detection)

AsmRAG is a novel malware detection framework that shifts the paradigm from traditional "black-box" probabilistic classification to evidence-based semantic retrieval. By leveraging Retrieval-Augmented Generation (RAG) at the assembly level, the system provides not only high-accuracy detection but also human-readable forensic explanations.

1. The Problem: The "Black Box" and "Syntactic Noise"

Current state-of-the-art malware detectors (e.g., EMBER, ResNeXt) suffer from two critical flaws:

Lack of Interpretability: They provide a probability score (e.g., "99% malicious") without explaining why. This leaves Security Operations Center (SOC) analysts without the forensic context needed for attribution.
Syntactic Fragility: Most models rely on global statistical features (byte histograms, entropy, or image-like textures). Adversaries exploit this by using obfuscation (dead-code insertion, register swapping, instruction substitution) to alter the file's "surface" statistics while keeping the underlying malicious logic intact. This pushes the malware across the model's decision boundary, causing evasion.

2. Methodology: Assembly-Level RAG

AsmRAG addresses these issues by treating detection as a Nearest Neighbor Retrieval problem on a semantic manifold. The pipeline consists of three main stages:

Stage 1: Knowledge Base (KB) Construction

Semantic Library Filtering: To prevent "noise" from benign, statically linked libraries (which can cause false positives), the system uses a specialized embedding model to identify and filter out boilerplate code.
Minimal Canonicalization: Unlike other models that strip all operands, AsmRAG preserves register names and immediate values (essential for identifying cryptographic constants) while only neutralizing randomized memory addresses (ASLR).
Embedding: A code-specialized LLM (Qwen3-Embedding) converts assembly functions into dense semantic vectors.

Stage 2: Evidence-Based Analysis

Distance-Weighted Voting: For a suspicious function, the system retrieves its $k$ -nearest neighbors from the KB. Instead of simple majority voting, it uses a scoring mechanism where closer semantic matches have higher influence.
Density-Weighted Anchor Selection: The system identifies the "Anchor Function" ( $q^*$ )—the specific code segment that is most centrally located within a known malware family's semantic cluster. This prevents the system from being distracted by common utility functions.

Stage 3: Generative Reporting & Active Learning

Differential Explanation: The system uses a generative LLM to compare the Anchor Function against a known "proof" sample. It produces natural language explanations (e.g., "Both functions implement a ROR-13 hash for API resolution") that highlight functional identity despite syntactic differences.
Active Learning Loop: When an analyst confirms a detection, the new function is immediately added to the KB, allowing the system to adapt to new threats without full retraining.

3. Key Contributions

Assembly RAG Architecture: A shift from global file-level statistics to atomic, function-level semantic retrieval.
Noise-Resilient Embedding: A strategy that preserves critical forensic artifacts (registers/constants) while remaining resilient to instruction substitution.
Anchor Function Attribution: A mathematical mechanism to isolate the most representative malicious logic for explanation.
Forensic Explainability: The ability to translate high-dimensional vector similarity into human-readable, verifiable evidence.

4. Results and Evaluation

The framework was tested on a curated dataset of 40,814 binaries across 17 malware families.

Detection Performance: Achieved a 96% F1-score, outperforming industry baselines like EMBER and ResNeXt.
Attribution Accuracy: Achieved a 95% F1-score for malware family identification.
Robustness (Adversarial Resilience): In a "Leave-One-Optimization-Out" test (testing against binaries compiled with different optimization levels like -O3), AsmRAG showed minimal degradation ( $\Delta < 3\%$ ), whereas EMBER and ResNeXt saw drops of up to 18%.
Explanation Quality: An automated audit (using Claude Opus 4.5) and human expert validation confirmed that 91.1% of generated claims were "Supported" by the actual assembly code, with a very low hallucination rate (2.6%).

5. Significance

AsmRAG represents a significant advancement in Explainable AI (XAI) for cybersecurity. By focusing on the intent of the code (semantics) rather than its appearance (syntax), it provides a defense that is naturally resilient to metamorphic obfuscation. Furthermore, its ability to provide "Atomic Evidence" transforms the role of the security analyst from a manual investigator to a rapid verifier, significantly increasing the speed and reliability of threat response.