Prompt to Pwn: Automated Exploit Generation for Smart Contracts

1. Problem Statement

Smart contracts are immutable once deployed, making post-deployment patching impossible. While Large Language Models (LLMs) have shown promise in detecting vulnerabilities, their ability to perform Automated Exploit Generation (AEG)—synthesizing executable Proof-of-Concept (PoC) exploits that are validated under real execution conditions—remains largely unexplored.

The authors identify a critical gap: detection tools (like Slither or Mythril) often produce false positives/negatives and do not confirm if a vulnerability is actually exploitable. Conversely, AEG can confirm severity and assess the risk of "script kiddies" using accessible LLM APIs to launch real-world attacks. The paper addresses the lack of systematic evaluation on whether frontier LLMs can autonomously generate, compile, and validate working exploits against smart contracts.

2. Methodology: The ReX Framework

To bridge this gap, the authors propose ReX, an execution-grounded, end-to-end framework that integrates LLM-based synthesis with the Foundry testing stack (a Rust-based toolchain for Solidity).

ReX Workflow:

1. Preprocessing: Cleans input contracts (removing comments/noise) and ensures compatibility (e.g., migrating to Solidity 0.8.26 for Foundry).
2. Script Generation: The LLM generates two scripts based on a prompt:
   • An Exploit Contract designed to trigger the vulnerability.
   • A Test Contract to validate the exploit against specific safety invariants.
3. Optional Optimization: Automated post-processing fixes common LLM errors (e.g., EIP-55 address checksums, missing payable casts).
4. Compilation & Testing: The scripts are compiled and executed via forge build and forge test.
5. Iterative Feedback Loop: If compilation or testing fails, error logs are fed back to the LLM to regenerate and repair the code. This loop continues until a valid exploit is found or a retry limit is reached.

Evaluation Datasets:

• SmartBugs-Curated: A standard benchmark of synthetic, labeled vulnerabilities used for controlled testing.
• Web3-AEG: A novel, curated dataset of 38+ real-world incident PoCs (2021–2025) involving high-impact DeFi hacks (e.g., Bybit, Pine Protocol). This dataset includes expert-written ground-truth exploits for validation.

Safety Invariants:
Instead of contract-specific rules, ReX uses class-level safety invariants (e.g., unauthorized balance changes, arithmetic overflows, unexpected reverts) to automatically determine if an exploit succeeded.

Models Evaluated:
Five frontier LLMs were tested: GPT-4.1, Gemini 2.5 Pro, Claude Opus 4, DeepSeek-R1, and Qwen3-Plus.

3. Key Contributions

1. First Systematic Empirical Study: The first unified benchmark and validation pipeline specifically for LLM-based AEG in smart contracts.
2. ReX Framework: A robust, automated pipeline that couples LLM synthesis with Foundry for deterministic validation.
3. Web3-AEG Dataset: The first public dataset of hand-written PoCs for real-world smart contract incidents, enabling realistic evaluation of AEG capabilities.
4. Defensive Strategies: Identification of specific hardening techniques that degrade LLM performance without necessarily fixing the underlying code (asymmetric defense).

4. Key Results

A. Performance on Benchmarks (Exp1)

• Gemini 2.5 Pro achieved the highest overall success rate (67.3%), followed by Claude Opus 4 (63.3%) and GPT-4.1 (58.1%).
• Vulnerability Types: LLMs performed best on Arithmetic (92.9% success for Gemini) and Bad Randomness vulnerabilities. They struggled with Reentrancy and Unchecked Low-Level Calls.
• Iterative Repair: The feedback loop significantly improved success rates (e.g., Reentrancy success jumped from 33% in 1 turn to 71% with 4 turns).

B. Real-World Performance (Exp2)

• Single-Contract vs. Cross-Contract: LLMs successfully generated working PoCs for single-contract vulnerabilities (e.g., predictable randomness, broken access control).
• Limitations: They failed to generate effective exploits for cross-contract attacks (e.g., complex flashloan reentrancy spanning multiple protocols). Human experts demonstrated superior ability to chain transactions across multiple contracts to maximize profit.
• Economic Gap: While LLMs could trigger technical invariants (e.g., minting tokens), they often lacked the "DeFi-aware" reasoning to chain these actions into profitable economic attacks (e.g., swapping tokens for ETH).

C. Factors Influencing Success (RQ2)

• Model Capability is Primary: The underlying LLM's general coding and reasoning ability (correlated with benchmarks like SWE-bench) was the strongest predictor of success.
• Contract Structure is Weak: Code complexity (nSLOC, nesting depth) showed weak correlation with exploitability. Complex contracts were not necessarily harder for LLMs to exploit than simple ones.
• Prompt Engineering: While structured prompts and role descriptions helped, they offered only marginal improvements (<5%) compared to the inherent capability of the model.

D. Defensive Implications (RQ3)

• Asymmetric Hardening: The study proposes defenses that specifically target LLM weaknesses:
  • Logic Splitting: Decomposing logic across multiple contracts forces LLMs to reason across boundaries, where they fail.
  • Decoy Vulnerabilities: Introducing false-positive patterns misleads LLMs.
  • Rare Constructs: Using inline assembly or uncommon syntax degrades LLM performance.
• Effectiveness: Combined hardening strategies reduced LLM success rates significantly (e.g., from ~87% to ~37-43%), though they did not eliminate the vulnerability itself.

5. Significance and Conclusion

The paper establishes that frontier LLMs are capable of automating the exploitation of simple, single-contract smart contract vulnerabilities, posing a tangible threat to the ecosystem. However, they currently lack the sophisticated reasoning required for complex, multi-contract DeFi attacks.

Key Takeaways:

• Threat Level: LLMs lower the barrier to entry for attackers, potentially enabling "script kiddies" to exploit common bugs (arithmetic, access control) rapidly.
• Defense Strategy: Traditional code obfuscation is less effective than structural complexity (logic splitting) and semantic obfuscation (decoys, rare constructs) which specifically disrupt LLM pattern matching.
• Future Work: There is a need for better evaluation metrics that account for economic viability (profit extraction) rather than just technical invariant violation, and for developing defenses that address the root causes of vulnerabilities rather than just hindering LLM analysis.

The study concludes that while LLMs are powerful tools for AEG, their current limitations in cross-contract reasoning offer a window for defense, provided developers adopt structural hardening techniques that specifically target LLM reasoning gaps.