A Multi-Agent Framework for Automated Exploit Generation with Constraint-Guided Comprehension and Reflection

1. Problem Statement

Modern software development relies heavily on open-source libraries, introducing significant security vulnerabilities. While static analysis tools can detect potential vulnerabilities at scale, they suffer from high false-positive rates, requiring extensive manual verification. Automated Exploit Generation (AEG) is a promising solution to confirm vulnerability authenticity by generating a working Proof-of-Concept (PoC). However, traditional AEG approaches face critical limitations:

• Fuzzing: Struggles with complex program logic and deep execution paths, failing to generate inputs that satisfy specific control-flow constraints.
• Symbolic Execution: Faces the "constraint-solving problem." Real-world code often involves complex string manipulations, higher-order functions, and polymorphism, which current SMT (Satisfiability Modulo Theories) solvers cannot effectively handle.
• Large Language Models (LLMs): While LLMs show potential for code generation, they face challenges in AEG:
  • Context Limitations: Vulnerability analysis often requires code spanning multiple files, exceeding context windows.
  • Hallucination & Focus: LLMs may lose focus or generate hallucinated code when dealing with complex, multi-step tasks.
  • Lack of Verification: LLMs cannot inherently verify if generated code is syntactically correct or logically sound without external feedback.

2. Methodology: VulnSage Framework

The authors propose VulnSage, a multi-agent framework that simulates the workflow of a human security researcher. It decomposes the AEG process into specialized sub-agents orchestrated by a central Supervisor Agent using an iterative ReAct (Reasoning and Acting) loop.

Core Components

1. Supervisor Agent:

   • Acts as the central controller.
   • Dynamically decides the next action (e.g., analysis, generation, validation, reflection) based on current progress and observations.
   • Terminates the process when an exploit is confirmed or a false positive is reasoned out.

2. Code Analyzer Agent:

   • Performs static analysis (taint flow analysis) on the target library.
   • Identifies vulnerabilities and extracts relevant code slices.
   • Key Output: Generates an alert containing the vulnerability type, dataflow details (callChainWithCtx), a set of relevant user-defined classes (inputClassSet), and an initial exploit template.

3. Code Generation Agent:

   • Constraint Extraction: Instead of formal SMT constraints, it extracts constraints in natural language by iteratively analyzing the call chain. It accumulates constraints (e.g., "the first argument must be a subclass of Job") to guide the LLM.
   • Exploit Generation: Uses a Chain-of-Thought (CoT) prompt to generate candidate exploit code that satisfies the extracted natural language constraints and the exploit template.

4. Validation Agent:

   • Executes the generated exploit code in a sandboxed environment.
   • Oracles: Uses specific checks based on vulnerability type (e.g., checking for command execution, JNDI lookups, or prototype pollution).
   • Anti-Cheat: Performs static and dynamic checks to ensure the exploit isn't cheating (e.g., directly calling Runtime.exec instead of triggering the vulnerability).
   • Feedback Collection: Captures compilation errors, runtime traces, and call stacks.

5. Reflection Agents:

   • Correction Insight Agent: Analyzes execution failures and runtime errors to generate "insights" (refined constraints) to correct the exploit code in the next iteration.
   • False Positive (FP) Reasoning Agent: Analyzes whether the failure is due to the alert being a false positive (e.g., presence of a sanitizer function or imprecise static analysis) and reports this to the Supervisor to terminate the loop early.

3. Key Contributions

1. Multi-Agent Architecture: A novel framework that overcomes LLM context limits and hallucination by decomposing AEG into specialized agents (Analyzer, Generator, Validator, Reflector) managed by a Supervisor.
2. Constraint-Guided Comprehension: A method to convert complex taint-flow analysis into natural language constraints. This allows LLMs to understand and satisfy complex logic (like polymorphism and string parsing) that SMT solvers cannot handle, without requiring formal constraint encoding.
3. Environment Feedback & Reflection: A continuous loop where execution traces and runtime errors are fed back to the LLM to refine the exploit or reason about false positives, mimicking human debugging.
4. Empirical Validation: Extensive evaluation showing superior performance over state-of-the-art tools and the discovery of real-world zero-day vulnerabilities.

4. Experimental Results

The framework was evaluated on SecBench.js (a benchmark of 475 vulnerable npm packages) and a Real-world Dataset (59k JS and 80k Java packages).

• Effectiveness (RQ1):
  • VulnSage generated 34.64% more exploits than the state-of-the-art tool EXPLOADE.js (88 more exploits).
  • It outperformed NodeMedic-FINE by 89.76% (228 more exploits).
  • On SecBench.js, VulnSage successfully generated exploits for 53.47% of vulnerabilities.
• Real-World Impact (RQ2):
  • Discovered 146 zero-day vulnerabilities in real-world packages.
  • 73 of these have been assigned CVE IDs, and 22 have been publicly disclosed.
  • Successfully handled complex constraints (e.g., specific string formats, polymorphic class instantiation) that caused other tools to fail.
• Component Analysis (RQ3):
  • MiniAlert (No Static Analysis Details): Success rate dropped significantly (e.g., to 47.71% on real-world data), proving that providing static analysis context is crucial.
  • NoRefl (No Reflection): Success rate dropped to ~69%, highlighting the importance of iterative refinement.
  • NoTrace (No Execution Path): Performance remained relatively high (~81%), suggesting that while execution traces are helpful, error messages alone are often sufficient for correction.
• Model Performance (RQ4):
  • Qwen3-Max performed best, attributed to its advanced architecture and large context window (252K tokens). Newer models generally outperformed older ones.
• Cost (RQ5):
  • Average time per alert: ~458 seconds (8 minutes).
  • Average cost per alert: ~$0.97 (using Qwen3-Max pricing).
  • The cost is considered acceptable for high-value security assessments.

5. Significance

VulnSage represents a paradigm shift in Automated Exploit Generation by moving away from purely formal constraint solving (SMT) or blind fuzzing toward a human-like, iterative, multi-agent reasoning process.

• Bridging the Gap: It effectively bridges the gap between static analysis (which finds many false positives) and dynamic validation (which is hard to automate).
• Scalability: By using natural language constraints and multi-agent decomposition, it handles the complexity of real-world object-oriented code (polymorphism, complex constructors) that stumps traditional tools.
• Practical Utility: The discovery of 146 zero-day vulnerabilities demonstrates that the framework is not just a theoretical exercise but a practical tool for securing software supply chains.
• Cost-Effectiveness: The ability to automate the verification of static analysis alerts reduces the manual burden on security teams, allowing them to focus on critical issues rather than filtering false alarms.

In conclusion, VulnSage demonstrates that combining static analysis with LLM-driven multi-agent reflection is a highly effective strategy for confirming vulnerabilities and generating exploits in complex, real-world software ecosystems.