Technical Summary: PoCGen

Problem Statement

Security vulnerabilities in software packages, particularly within the npm ecosystem, pose significant risks to developers and users. While timely patching is crucial, the vulnerability reporting process often lacks Proof-of-Concept (PoC) exploits. These exploits are essential for developers to understand the vulnerability, verify patches, and prevent regressions. However, creating a PoC is challenging because vulnerability reports are often informal, incomplete, and lack specific details about which functions are vulnerable or what inputs trigger the issue.

Existing automated approaches have limitations:

• Large Language Models (LLMs) alone lack the necessary context about specific codebases and have limited reasoning capabilities regarding code behavior.
• Traditional analysis tools (e.g., fuzzing, symbolic execution like Explode.js) often fail to leverage the information provided in informal vulnerability reports and struggle with the complexity of the npm ecosystem's dependencies.

Consequently, many vulnerability reports (e.g., in the SecBench.js dataset) remain without executable PoC exploits, hindering effective remediation.

Methodology: PoCGen

PoCGen is a novel approach that autonomously generates and validates PoC exploits for npm packages by combining LLMs, static analysis, and dynamic analysis. The system operates through four iteratively executed components:

1. Vulnerability Information Extraction:

   • Vulnerability Type & Function: An LLM analyzes the informal report to identify the vulnerability type (e.g., Path Traversal, Prototype Pollution, Command Injection, Code Injection, ReDoS) and ranks potential vulnerable functions.
   • Taint Path Extraction: Static taint analysis (using CodeQL) traces data flow from the identified candidate functions to vulnerable sinks. If standard analysis fails, the system employs extended rules or falls back to dynamic analysis combined with LLM guessing.
   • Usage Snippets: The system extracts code examples of the vulnerable function from source code and documentation to provide context on function signatures and setup requirements.

2. Exploit Generation:

   • The system constructs a prompt for the LLM containing the vulnerability description, the vulnerable function, usage snippets, similar exploit examples (few-shot learning), and the extracted taint paths.
   • The LLM is tasked with generating a JavaScript file that triggers a specific security-relevant action (e.g., executing a specific command or accessing a flag file).

3. Exploit Validation:

   • Generated candidates are executed in a controlled environment.
   • Vulnerability-Specific Checkers: Validators verify if the exploit triggers the expected effect (e.g., for ReDoS, execution time exceeds 1,500ms; for Command Injection, a specific file is created).
   • Sanity Checks: The system ensures the exploit actually calls the vulnerable function (preventing false positives where the LLM bypasses the function) and verifies the exploit matches the report description.

4. Prompt Refinement:

   • If validation fails, PoCGen enters a refinement loop. It uses refiners to provide additional context to the LLM:
     • Context Refiners: Provide full function bodies or definitions of missing variables/functions found in the taint path.
     • Runtime Refiners: Feed back runtime errors, code coverage information (identifying unexecuted paths), debugger values, and specific values passed to sinks.
   • The prompt is iteratively updated until a valid exploit is found or the computational budget (time, tokens, iterations) is exhausted.

Key Contributions

1. Novel Approach: A framework that autonomously generates and validates PoC exploits for npm packages by integrating LLMs with static and dynamic analysis techniques.
2. Empirical Evaluation: Comprehensive evaluation on the SecBench.js dataset (560 real-world vulnerabilities) and a set of 126 recent real-world vulnerabilities from GitHub Security Advisories.
3. Resource Sharing: The authors have released the source code, datasets, and experiment scripts to foster future research.

Results

• Success Rate: PoCGen successfully generated PoC exploits for 71% (395 out of 560) of vulnerabilities in the SecBench.js dataset.
• Comparison: This performance significantly outperforms the state-of-the-art tool Explode.js (32% success rate) by 38 absolute percentage points. It also outperformed an LLM-based agent (Mini-SWE-Agent) by 42 percentage points.
• Real-World Application: On 126 recent vulnerabilities (reported after the LLM's knowledge cutoff), PoCGen generated successful exploits for 60.3%.
• Impact: Five of the generated exploits were accepted and added to the GitHub Security Advisories database, augmenting reports that previously lacked PoCs.
• Cost Efficiency: The average cost per generated exploit is approximately $0.02, with an average generation time of 7 minutes for successful attempts.

Significance and Claims

The paper positions PoCGen as a practical tool to bridge the gap between vulnerability discovery and remediation. Its significance lies in:

• Accelerating Patching: By automating PoC generation, developers can better understand vulnerabilities and verify patches more rapidly.
• Improving Regression Testing: Generated exploits can be directly integrated into test suites to prevent future regressions.
• Enhancing Vulnerability Reports: The approach can augment existing reports with missing PoCs, improving the quality of security advisories.
• Cost-Effectiveness: The low computational cost makes the approach viable for practical adoption by security teams and researchers.

The authors emphasize that while the approach targets "proof-of-concept" exploits intended for testing and demonstration, it provides a mechanism to improve the security posture of the npm ecosystem by making vulnerabilities more actionable and verifiable.