💻 computer science

PoCEvolve: Generating Proof-of-Concept Exploits from Security Patches with Vulnerability-Aware Prompt Evolution

PoCEvolve is a vulnerability-aware prompt-evolution framework that generates proof-of-concept exploits directly from security patches by iteratively refining prompts based on inferred vulnerability context, significantly outperforming existing methods in scenarios where detailed vulnerability reports are unavailable.

Original authors: Duc Manh Tran, Ratnadira Widyasari, Ivana Clairine Irsan, Huihui Huang, Ting Zhang, Shar Lwin Khin, Ouh Eng Lieh, Hong Jin Kang, David Lo

Published 2026-07-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Duc Manh Tran, Ratnadira Widyasari, Ivana Clairine Irsan, Huihui Huang, Ting Zhang, Shar Lwin Khin, Ouh Eng Lieh, Hong Jin Kang, David Lo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the digital world as a massive, bustling city where software programs are the buildings. Sometimes, a crack appears in a wall—a "vulnerability"—that a burglar could use to sneak in. Usually, the building's architects (the developers) fix the crack and post a notice saying, "We found a hole here, and here is exactly how it happened." But in the real world, the fix often gets posted first, while the detailed explanation arrives days, weeks, or even months later. This creates a dangerous "blind spot": the burglar sees the patch, figures out how to break in, and strikes, while the building's security guards are still waiting for the manual to understand the threat. To bridge this gap, security experts need a "Proof-of-Concept" (PoC). Think of a PoC not as a weapon, but as a practice lockpick. It's a harmless, test-run demonstration that proves, "Yes, if you try this specific move, the lock opens." It helps defenders know exactly which doors are broken and how to secure them before the bad guys do.

For a long time, computers struggled to make these practice lockpicks on their own. They needed a detailed manual (a vulnerability report) to know what to do. But what if the manual isn't there yet? What if the only clue is the patch itself? This is the puzzle tackled by a new tool called POCEVOLVE. The researchers behind it asked: "Can we teach an AI to look at a patch, figure out the secret weakness, and build a working practice exploit all by itself, even without the manual?" They didn't just guess; they built a system that acts like a tireless, curious student. When the AI tries to build a lockpick and fails, instead of giving up, it looks at why it failed. Did it use the wrong tool? Was the key too short? Did it forget to turn the handle? The system takes these mistakes, learns from them, and tweaks its instructions for the next try. It's like a chef tasting a soup that's too salty, realizing they added too much salt, and adjusting the recipe for the next batch, all without ever having seen the original recipe book.

The paper reveals that this "learning from failure" approach works surprisingly well. The team tested POCEVOLVE on 190 real-world software vulnerabilities in the JavaScript ecosystem (a popular language for web apps). They found that when the AI was allowed to evolve its own instructions based on its mistakes, it could successfully generate a working practice exploit for 58.4% of the vulnerabilities using a standard AI model, and up to 85.3% with a more advanced model. This is a significant jump compared to older methods that couldn't adapt. For instance, with the standard model, the success rate went from 48.4% (using the old method) to 58.4% (using POCEVOLVE). Even more impressively, when they tried to generate exploits for a specific type of security hole called "Command Injection," the old method failed completely (0%), while POCEVOLVE succeeded in nearly half of those cases.

The researchers also checked how much this "learning" costs. They found that while the process takes a bit more time and money (ranging from about $0.03 to $0.25 per vulnerability depending on the AI model used), it is still fast enough to be useful. In fact, the average time it takes to solve one vulnerability is between 5 minutes and 52 seconds and 52 minutes and 54 seconds. This is much faster than the average gap between a patch being released and a detailed report appearing, which the study found to be 18 days for half of the cases they looked at. This suggests that defenders could theoretically use this tool to get their "practice lockpicks" ready almost immediately after a patch drops, closing the blind spot before attackers can exploit it.

However, the paper is careful to note that this isn't a magic wand that solves everything. The system still struggles if the AI gets confused about which part of the code is actually broken, or if the "story" it builds up over many attempts becomes too long for the AI to remember. The results are based on simulations and tests on specific software libraries, so while the numbers are promising, they represent what happened in this specific experiment, not a guarantee for every software problem in the world. The authors suggest that this approach is a powerful new way to handle security, but it works best when combined with human oversight and when the AI is given the right kind of feedback to learn from. Ultimately, POCEVOLVE suggests that by teaching computers to learn from their own mistakes, we can stay one step ahead of the burglars, even when the blueprints for the break-in haven't been written yet.

Technical Summary: POCEVOLVE

1. Problem Statement

The paper addresses a critical gap in the vulnerability remediation lifecycle: the "patch-only" window. Empirical analysis of 26,803 CVEs reveals that 97.1% of security fixes are committed before public disclosure, with a median gap of 18 days and a mean of 147 days. During this interval, patches are public, allowing attackers to reverse-engineer vulnerabilities, yet defenders lack detailed vulnerability reports to assess exposure or validate fixes. Furthermore, over 70% of these patches lack accompanying test files.

Existing state-of-the-art tools, such as POCGEN, rely on detailed vulnerability reports to generate Proof-of-Concept (PoC) exploits. They cannot operate solely on vulnerability-fixing commits because commit messages and diffs often omit critical details like the root cause, affected APIs, or trigger conditions. The authors formulate a new problem setting: generating executable PoC exploits directly from vulnerability-fixing commits without access to ground-truth vulnerability reports.

2. Methodology: POCEVOLVE

The authors propose POCEVOLVE, a vulnerability-aware prompt-evolution framework designed to synthesize PoC exploits from commits. The system operates in three phases:

Phase 1: Commit Analyzer

Given a vulnerability-fixing commit ( $C$ ) and repository context ( $R$ ), an LLM reconstructs a structured vulnerability report. This includes:

Vulnerability Type: Classified into five categories (Path Traversal, Prototype Pollution, Command Injection, Code Injection, ReDoS).
Vulnerable API: Identification of the specific function or module affected.
Natural Language Description: A synthesized description of the vulnerability based on the commit message and code diff.

Phase 2: Exploit Generator

The generator (defaulting to POCGEN or a direct LLM) uses the reconstructed report to produce candidate PoC exploits. If a candidate fails validation, the system does not simply retry; it enters a feedback loop.

Phase 3: Prompt Evolver

This is the core innovation. Instead of relying solely on runtime execution feedback (which POCGEN uses via fixed hints), POCEVOLVE introduces a vulnerability-context-guided scoring mechanism.

Dimension Scoring: When an exploit fails, the system evaluates the prompt against eight specific vulnerability-context dimensions:
1. Candidate Vulnerable API
2. Vulnerability Description
3. Usage Snippets
4. Exploit Skeleton
5. Similar Exploits
6. Taint-path Snippets
7. Debugger Output
8. Code Coverage
Feedback Generation: An LLM assigns a score ( $[0, 1]$ ) to each dimension to determine its usefulness in diagnosing the failure and generates natural language feedback (e.g., "The payload structure is incorrect; add the missing callback").
Pareto Selection: The system maintains a pool of failed prompt candidates. It applies a Pareto-based selection strategy to retain non-dominated prompts that offer strong trade-offs across the eight dimensions, ensuring diversity in the search space.
Evolution: The selected prompts are synthesized into a new, improved prompt for the next generation iteration. This loop continues until a valid exploit is found or the budget (5 iterations) is exhausted.

3. Key Contributions

Problem Formulation & Empirical Study: The authors define the "patch-only" PoC generation setting and provide empirical evidence (26,803 CVEs) showing its prevalence, highlighting the 18-day median gap between patching and disclosure.
POCEVOLVE Framework: A novel framework that leverages vulnerability-context-guided prompt evolution. Unlike general prompt evolution, it uses task-specific criteria (the 8 dimensions) to assess and refine prompts based on the specific nature of security vulnerabilities.
Extensive Evaluation: The system is evaluated on SecBench.VFC.js (190 real-world vulnerabilities with fixing commits) and SecBench.js (with reports), demonstrating effectiveness across different LLMs (GPT-4o-mini, Qwen3.7-Plus) and generator configurations.

4. Experimental Results

The evaluation was conducted on a benchmark of 190 JavaScript vulnerabilities (SecBench.VFC.js).

Performance with GPT-4o-mini:
- Direct LLM Prompting: 19.5% success rate.
- POCEVOLVE (LLM): 28.4% success rate (+45.9% relative improvement).
- POCGEN (Baseline): 48.4% success rate.
- Full POCEVOLVE: 58.4% success rate (+20.7% over POCGEN; +200.0% over direct LLM).
Performance with Qwen3.7-Plus:
- Full POCEVOLVE: 79.5% success rate (vs. 64.2% for POCGEN, a +23.8% improvement).
- POCEVOLVE (LLM): 85.3% success rate.
Report-Guided Setting: Even when detailed vulnerability reports are available, POCEVOLVE improves over POCGEN by 11.1% (71.7% vs. 60.6%), indicating that prompt evolution adds value beyond the initial report.
Cost and Runtime:
- The average cost per evaluated vulnerability ranges from $0.0292 (POCEVOLVE with GPT-4o-mini) to $0.2492 (Full POCEVOLVE with Qwen3.7-Plus).
- Runtime per vulnerability ranges from ~6 minutes (GPT-4o-mini) to ~53 minutes (Qwen3.7-Plus). The authors note these runtimes are significantly shorter than the median 18-day patch-to-disclosure window.

5. Significance and Claims

The paper claims that POCEVOLVE successfully bridges the information gap during the patch-only window, enabling defenders to generate executable evidence (PoCs) before detailed reports are available.

Feasibility: The results demonstrate that vulnerability-fixing commits and repository context contain sufficient information to reconstruct exploit logic, challenging the assumption that detailed reports are a prerequisite for automated exploit generation.
Generalizability: The prompt evolution strategy improves performance across different underlying LLMs and generator architectures, suggesting that the vulnerability-context scoring mechanism is a robust enhancement to automated security analysis.
Practicality: Despite the added computational cost of prompt evolution, the total runtime remains well within the typical delay between patching and disclosure, making the approach viable for real-world defensive operations.

The authors conclude that future exploit generators should adopt this patch-only setting and that the proposed framework offers a practical method for accelerating vulnerability assessment and validation.