CVE-Factory: Scaling Expert-Level Agentic Tasks for Code Security Vulnerability
This paper introduces CVE-Factory, a multi-agent framework that automatically transforms sparse CVE metadata into high-quality, executable security tasks to create the LiveCVEBench benchmark and a large-scale training dataset, significantly improving the vulnerability detection capabilities of code agents.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are a security guard trying to teach a new robot how to fix broken locks on a million different doors. The problem is, the only instructions you have are tiny, vague sticky notes that say, "Door 404 is broken," without telling you how it's broken, what the door looks like, or what tools you need to fix it.
This is the current state of AI security research. We have lists of vulnerabilities (called CVEs), but they are just sparse data points. To train AI agents to fix them, we need full "training kits": a working model of the broken door, a test to prove it's broken, and a verified solution.
CVE-Factory is a new system that acts like a super-efficient, automated construction crew. It takes those tiny, vague sticky notes and instantly builds a complete, working training kit for the robot.
Here is how it works, using simple analogies:
1. The Problem: The "Vague Sticky Note"
Currently, security experts have to manually build these training kits. They read a vulnerability report, find the software, set up a fake computer environment, break it on purpose, write a test to catch the break, and then write the fix.
- The Reality: This takes experts 10+ hours per door. It's too slow and expensive to do for thousands of doors.
- The Result: AI robots don't have enough practice material, so they remain bad at security.
2. The Solution: The "Assembly Line" (CVE-Factory)
The authors built CVE-Factory, which is like a specialized factory with a conveyor belt. Instead of one person doing the whole job, they broke the process down into six distinct stations, each run by a specialized AI agent.
Think of it like a high-tech car repair shop where no single mechanic tries to do everything at once:
- Station 1: The Detective (Analyzer)
The AI reads the vague sticky note and goes on a web search to find the blueprints, the specific model of the car, and the exact part that's broken. It creates a clear "case file." - Station 2: The Architect (Generator)
Using the case file, this AI writes the instructions for the robot: "Here is the broken door. Here is a test to prove it's broken. Here is what a perfect fix looks like." - Station 3: The Builder (Builder)
This AI builds the actual environment. It sets up the "fake computer" (using Docker containers) to look exactly like the real software. Crucially, it builds this without seeing the solution or the test, so it doesn't cheat. It just builds the scene. - Station 4: The Inspector (Validator)
Before anyone tries to fix it, this agent checks: "Is the door actually broken in this fake room? Does the test work?" If the room isn't set up right, it sends it back to the Builder to fix. - Station 5: The Mechanic (Solver)
This agent tries to fix the door using the instructions. It applies the patch and runs the test. - Station 6: The Quality Control Chief (Checker)
The final boss. It runs a full end-to-end check. Did the fix actually work? Did the robot accidentally break something else? Is the test real, or did the robot just fake the results?
The Secret Sauce: If something goes wrong at any station, the system doesn't just give up. It has a "feedback loop." If the Inspector says, "The door isn't broken," it sends the task back to the Builder to rebuild the door, not to a new person. This ensures the final product is high-quality.
3. The Results: Expert-Level Speed
The team tested this factory against human experts who had already built 215 of these training kits.
- Accuracy: The factory's kits were 95% as good as the human experts' kits.
- Speed: Humans take 5–24 hours per kit. The factory takes about 48 minutes.
- Scale: With 20 workers running at once, the factory built all 215 kits in under 5 hours. A human team would have taken weeks.
4. The New Playground: LiveCVEBench
Because the factory is so fast, the authors didn't just stop at 215. They built a new, constantly updating playground called LiveCVEBench.
- It contains 190 real-world security tasks.
- It covers 14 different programming languages (not just Python).
- It includes emerging threats, like vulnerabilities in AI tools themselves.
- It updates automatically as new vulnerabilities are discovered, unlike old benchmarks that are frozen in time.
5. Training the Robot
Finally, they used the factory to create over 1,000 training tasks to teach an AI model (Qwen3-32B).
- Before training: The AI could solve only 5.3% of the security tasks.
- After training: It jumped to 35.8%, beating much larger, more expensive models.
- Bonus: The skills it learned weren't just for security; it got better at general coding tasks too.
Summary
CVE-Factory is a multi-agent system that automates the creation of high-quality security training data. It turns sparse, boring vulnerability reports into full, interactive "escape room" style challenges for AI agents. It proves that by breaking a complex job into smaller, specialized steps and letting AI agents collaborate, we can generate expert-level training data at a scale and speed that humans simply cannot match. This allows us to train smarter, more secure AI agents faster than ever before.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.