🤖 AI

MulVul: Retrieval-augmented Multi-Agent Code Vulnerability Detection via Cross-Model Prompt Evolution

MulVul is a retrieval-augmented multi-agent framework that addresses the limitations of single-model vulnerability detection and manual prompt engineering by employing a coarse-to-fine routing strategy and a novel cross-model prompt evolution mechanism to achieve significantly higher accuracy across diverse vulnerability types.

Original authors: Zihan Wu, Jie Xu, Yun Peng, Chun Yong Chong, Xiaohua Jia

Published 2026-01-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zihan Wu, Jie Xu, Yun Peng, Chun Yong Chong, Xiaohua Jia

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find specific types of bugs in a massive library of computer code. This is a bit like trying to find a needle in a haystack, but the "needles" come in thousands of different shapes, sizes, and colors.

The paper introduces a new system called MulVul to solve this problem. Here is how it works, explained through simple analogies:

The Problem: One Size Doesn't Fit All

Imagine you hire a single, very smart detective to find every kind of crime in a city.

The Issue: Some crimes are like bank robberies (requiring knowledge of locks and alarms), while others are like identity theft (requiring knowledge of social engineering). A single detective might be great at one but terrible at the other.
The Old Way: Previous AI tools tried to use one "super-detective" to catch everything at once. Because there are so many different types of code vulnerabilities (over 130 types!), this single detective often gets confused, misses things, or cries wolf (false alarms).
The Manual Problem: You could try to write a specific set of instructions (a "prompt") for the AI for every single type of vulnerability. But with hundreds of types, writing and testing these instructions by hand is impossible—it takes too long and is too expensive.

The Solution: MulVul (The Specialized Team)

MulVul changes the game by using a team of specialists instead of one generalist. It works in two main phases:

1. The "Router" (The Traffic Cop)

When a piece of code arrives, it doesn't go to everyone. First, it goes to a Router Agent.

What it does: The Router looks at the code and asks, "What kind of trouble is this likely to be?" It doesn't need to know the exact bug; it just needs to guess the broad category (e.g., "This looks like a memory error" or "This looks like an injection attack").
The Trick: It uses a Retrieval Tool. Think of this as the Router flipping through a giant, organized encyclopedia of past crimes to find similar cases before making a guess. This helps it avoid guessing randomly.
The Result: It picks the top 3 most likely categories and sends the code only to the specialists who handle those specific categories. This saves time and money.

2. The "Detectors" (The Forensic Experts)

Once the Router sends the code to the right specialists, the Detector Agents take over.

What they do: These are highly specialized agents. One only looks for memory errors; another only looks for injection attacks.
The Trick: Like the Router, they also use a Retrieval Tool. But they are smarter about it. They look for "contrastive" examples. Imagine a detective comparing a suspect's story to both a true story of a similar crime and a false story of a similar crime to spot the tiny differences. This helps them pinpoint the exact bug without getting confused by similar-looking code.
Isolation: Crucially, these detectives work alone. They don't talk to each other. If one detective makes a mistake, it doesn't spread to the others, preventing a chain reaction of errors.

The Secret Sauce: "Cross-Model Prompt Evolution"

Writing the instructions (prompts) for these AI agents is the hardest part. If you ask one AI to write instructions for itself, it might get stuck in a loop, thinking its own bad ideas are good.

MulVul uses a clever two-AI system to fix this:

The Generator (The Architect): One AI (like Claude) tries to write and improve the instructions. It guesses, "Maybe if I say it this way, the AI will do better."
The Executor (The Tester): A different AI (like GPT-4o) actually tries to follow those instructions to find bugs. It reports back: "That instruction worked well" or "That one failed."
The Loop: The Generator uses this feedback to write better instructions, and the Tester checks them again. Because they are different models, the "Architect" can't trick the "Tester" into thinking a bad idea is good. This creates a cycle of constant improvement until the instructions are perfect.

The Results

The authors tested this system on a massive dataset of real-world code vulnerabilities (called PrimeVul).

The Score: MulVul achieved a score of 34.79%, which is 41.5% better than the best previous method.
Why it matters: It found more real bugs and made fewer false alarms than anything else tested.
The "Evolution" Boost: The part where the two AIs improved the instructions together made the system 51.6% better than if humans had just written the instructions manually.

Summary

MulVul is like a high-tech security firm that doesn't rely on one overworked guard. Instead, it uses a smart traffic cop to direct suspects to the right forensic expert, and those experts use a library of past cases to solve the puzzle. To make sure the experts know exactly what to do, the firm uses two different AI brains to constantly rewrite and perfect their rulebooks, ensuring they catch the most dangerous bugs with high accuracy.

Technical Summary: MulVul

Problem Definition

The paper addresses two critical limitations in automating code vulnerability detection using Large Language Models (LLMs):

Heterogeneity of Vulnerability Patterns: Vulnerability types (e.g., buffer overflows vs. injection attacks) require distinct reasoning capabilities. A single unified model struggles to capture these diverse, type-specific patterns within a shared latent space, leading to missed vulnerabilities or high false alarm rates.
Scalability of Prompt Engineering: Real-world systems involve hundreds of Common Weakness Enumeration (CWE) entries. Manually engineering unique prompts for specialized agents across such a vast number of categories is unfeasible. Furthermore, relying on a single model for both prompt generation and evaluation introduces self-correction bias, while multi-agent systems risk amplifying hallucinations if not properly grounded.

Existing methods often rely on fixed prompts without external grounding or attempt to detect all vulnerability types simultaneously with a single model, failing to balance computational efficiency with broad coverage.

Methodology: MulVul Framework

MulVul is a retrieval-augmented multi-agent framework designed for precise and broad-coverage vulnerability detection. It operates through a coarse-to-fine strategy and employs Cross-Model Prompt Evolution.

1. Architecture: Router-Detector

The system utilizes a hierarchical architecture aligned with the CWE taxonomy:

Router Agent: Acts as a dispatcher. It first predicts the top- $k$ coarse-grained vulnerability categories (e.g., "Memory Buffer Errors"). It uses a Global Retrieval Tool to query a knowledge base for cross-category evidence, ensuring the correct category is not filtered out early.
Detector Agents: Specialized agents are invoked only for the categories selected by the Router. Each Detector performs fine-grained identification of specific vulnerability types (e.g., "CWE-119 Buffer Overflow").
- Isolation: Detectors operate in isolation without inter-agent communication to prevent error cascading.
- Contrastive Retrieval: Each Detector uses a specialized tool to retrieve evidence from three pools: in-category positives, clean examples, and out-of-category "hard negatives." This contrastive approach helps distinguish subtle security flaws.

2. Knowledge Base (SCALE)

The framework relies on a vulnerability knowledge base constructed using SCALE (Structured Code Representation). SCALE converts code snippets into Structured Comment Trees (natural language comments attached to AST nodes encoding control flow). These representations are embedded (using UniXcoder) and indexed for efficient nearest-neighbor retrieval.

3. Cross-Model Prompt Evolution

To solve the scalability bottleneck of manual prompt engineering, MulVul introduces a novel optimization mechanism that decouples prompt generation from evaluation:

Generator LLM ( $M_{evo}$ ): (e.g., Claude) iteratively proposes and mutates candidate prompts.
Executor LLM ( $M_{exec}$ ): (e.g., GPT-4o) executes the agents with these prompts and returns performance feedback (Recall@k for the Router; F1 for Detectors).
Process: This separation mitigates the self-correction bias inherent in single-model optimization. The Router prompt is optimized for recall (ensuring the correct category is in the top- $k$ ), while Detector prompts are optimized for the precision-recall balance (F1 score).

Key Contributions

MulVul Framework: A novel retrieval-augmented multi-agent system that balances computational efficiency with detection coverage by routing inputs to specialized experts only when necessary.
Cross-Model Prompt Evolution: A mechanism that automatically optimizes prompts for specialized agents by separating generation and execution across different LLMs, effectively solving the scalability challenge of manual engineering and reducing bias.
State-of-the-Art Performance: Comprehensive experiments demonstrating significant improvements over existing baselines in multi-class vulnerability detection.

Experimental Results

The framework was evaluated on the PrimeVul benchmark, covering 6,968 vulnerable and 229,764 benign C/C++ functions across 130 CWE types.

Overall Performance: MulVul achieved a Macro-F1 of 34.79% at the type-level, outperforming the best baseline (LLM×CPG) by 10.21 percentage points (a 41.5% relative improvement).
Precision-Recall Trade-off: The system achieved a Macro-Precision of 27.90% and Macro-Recall of 46.19%, demonstrating a superior balance compared to baselines that often sacrifice one for the other.
Ablation Studies:
- Retrieval: Removing retrieval caused the largest performance drop (F1 fell from 34.56% to 21.80%), confirming the necessity of grounding reasoning with evidence.
- Prompt Evolution: Replacing evolved prompts with manual templates reduced performance by 11.76%, validating the effectiveness of the cross-model evolution strategy.
- Multi-Agent: Removing the agent architecture (using a single model) resulted in a 5.95% drop.
Few-Shot Performance: MulVul demonstrated robust performance on rare CWE types (fewer than 100 samples), achieving ~48% F1 compared to ~25% for the best baseline. This highlights the system's ability to transfer knowledge across related vulnerability patterns.
Routing Parameter ( $k$ ): The optimal routing parameter was found to be $k=3$ , where the Macro-F1 peaked at 34.79%.

Significance and Claims

The paper claims that MulVul represents a new state-of-the-art for multi-class vulnerability detection. Its significance lies in:

Addressing Heterogeneity: By decomposing the task into coarse routing and fine-grained detection, it overcomes the limitations of unified models in handling diverse vulnerability patterns.
Scalability: The Cross-Model Prompt Evolution mechanism automates the creation of specialized instructions for hundreds of categories, making broad-coverage detection feasible without manual engineering.
Robustness: The combination of retrieval-augmented generation and isolated agent operation effectively mitigates hallucinations and error amplification, which are common pitfalls in multi-agent LLM systems.

The authors acknowledge limitations, including evaluation restricted to C/C++ code, the computational cost of multiple LLM API calls, and the need for further evaluation of prompt transferability to other LLM backbones. They also note the potential dual-use risk of automated vulnerability detection tools.