EVMbench: Evaluating AI Agents on Smart Contract Security

The paper introduces EVMbench, a benchmarking framework that evaluates the capabilities of frontier AI agents in detecting, patching, and exploiting smart contract vulnerabilities within a realistic local Ethereum environment, revealing their ability to successfully execute end-to-end attacks against live blockchain instances.

The Gist

Imagine a world where money isn't kept in banks with security guards and vaults, but in digital glass houses called "Smart Contracts." These houses are built on a public blockchain (like Ethereum). Once the blueprints are drawn and the house is built, you can't change a single brick without everyone noticing. If the blueprints have a flaw, a thief can walk right in and steal everything, and because the house is on a public ledger, the theft is instant and impossible to reverse.

Right now, Artificial Intelligence (AI) is getting very good at reading blueprints and writing code. The big question is: Is the AI smart enough to find the flaws in these glass houses before bad guys do? Or is it smart enough to break in and steal the money itself?

This paper, EVMbench, is the answer to that question. It's a giant "test drive" for AI agents to see how they handle the high-stakes world of crypto security.

Here is a breakdown of the paper using simple analogies:

1. The Setting: The "Dark Forest"

Imagine the blockchain as a Dark Forest.

• The Trees: These are the smart contracts holding billions of dollars.
• The Hunters: These are hackers (and now, AI bots) scanning the forest. If they find a weak branch (a vulnerability), they can shake it until the fruit (money) falls into their basket.
• The Problem: In this forest, if you make a mistake, you lose everything instantly. There are no "undo" buttons.

2. The Test: EVMbench

The researchers built a giant training gym called EVMbench. Instead of just asking the AI to "write a poem" or "solve a math problem," they put the AI in a simulated version of this Dark Forest with 117 different "glass houses" that have known holes in them.

The AI has to play three different games:

Game A: The Detective (Detect Mode)

• The Task: The AI is given a stack of blueprints and asked to find all the holes.
• The Goal: Write a report saying, "Hey, this window is unlocked," or "This door has no lock."
• The Result: The best AI found about 46% of the hidden traps. It's getting better, but it's still missing a lot of the subtle cracks that human experts catch.

Game B: The Handyman (Patch Mode)

• The Task: The AI finds the holes and has to fix them without breaking the house.
• The Goal: Put a new lock on the door or reinforce the window, but make sure the house still works for the people living inside.
• The Result: The AI is surprisingly good at this. Once it knows where the problem is, it can often fix it perfectly. The main struggle isn't fixing the code; it's finding the problem in the first place.

Game C: The Thief (Exploit Mode)

• The Task: This is the scary part. The AI is given a fake wallet with some money and told: "Find a way to steal the money from these houses."
• The Goal: The AI has to figure out the exact sequence of moves to trick the house into giving up its funds.
• The Result: This is the most concerning finding. The AI is already capable of breaking into these houses end-to-end. In some tests, the top AI successfully stole the "money" (simulated tokens) in 71% of the attempts. It didn't just find the hole; it walked through it, emptied the vault, and ran away.

3. The "Flash Loan" Analogy

One of the coolest (and scariest) examples in the paper involves something called a Flash Loan.

• Imagine: You walk into a bank, borrow a billion dollars, use that money to buy a house, sell the house, pay back the billion dollars, and keep the profit—all in the time it takes to blink.
• The AI's Move: In the test, the AI figured out how to use a "Flash Loan" to trick a complex financial machine. It borrowed money, used it to trick the machine into thinking it was a trusted friend, stole the real money, and paid back the loan instantly. The AI did this entirely on its own, without a human telling it exactly how to do it.

4. Why This Matters

The paper concludes with a mix of hope and warning:

• The Good News: AI is getting really good at being a security guard. If we use AI to audit code, we might find bugs faster than humans can, making the financial system safer.
• The Bad News: AI is also getting really good at being a burglar. If a bad actor gives an AI a "hack button," it could potentially drain billions of dollars from crypto systems before humans even realize what's happening.

The Bottom Line

Think of EVMbench as a stress test for the future. It shows us that AI agents are no longer just chatbots that can write code; they are becoming autonomous agents that can understand complex systems, find weaknesses, and execute attacks (or defenses) in the real world.

The researchers are releasing their test data to the public so that security experts can keep training AI to be better defenders, ensuring that when the AI gets even smarter, it's on the side of the good guys, not the thieves in the Dark Forest.

Technical Summary

Here is a detailed technical summary of the paper "EVMbench: Evaluating AI Agents on Smart Contract Security."

1. Problem Statement

Smart contracts on public blockchains manage hundreds of billions of dollars in assets. Unlike traditional software, smart contracts are immutable and execute deterministically; vulnerabilities often lead to immediate, irreversible financial losses. As AI agents become more capable at reading, writing, and executing code, there is a critical need to evaluate their ability to navigate this high-stakes environment.

Current cybersecurity benchmarks (e.g., CVE-based or CTF challenges) fail to capture the full lifecycle of smart contract security, which involves:

1. Discovery: Finding subtle, high-severity bugs in complex, real-world codebases.
2. Mitigation: Patching vulnerabilities without breaking existing functionality.
3. Exploitation: Executing end-to-end attacks against live blockchain instances to drain funds.

Existing evaluations often lack realistic environments, rely on simplified subtasks, or do not measure the "comprehensive coverage" required to find all vulnerabilities in a codebase rather than just one.

2. Methodology: EVMbench

The authors introduce EVMbench, a comprehensive evaluation framework designed to measure AI agents on detecting, patching, and exploiting smart contract vulnerabilities.

A. Dataset Curation

• Source: 117 high-severity vulnerabilities curated from 40 distinct audit repositories, primarily sourced from Code4rena (a competitive smart contract auditing platform) and the Tempo blockchain security process.
• Selection Criteria: Only vulnerabilities that could directly lead to the loss of user or platform funds were included. The dataset includes a mix of access control issues, reentrancy, logic errors, and economic exploits.
• Ground Truth: Each task includes oracle patches (verified fixes) and oracle exploits (verified attack scripts) to serve as grading standards.

B. Evaluation Modes

EVMbench operates in three distinct modes, each with specific grading mechanisms:

1. Detect Mode:

   • Task: The agent audits a codebase and produces a report identifying high-severity vulnerabilities.
   • Grading: A model-based judge compares the agent's report against the ground-truth audit report. The score is the percentage of ground-truth vulnerabilities correctly identified (Recall).
   • Metric: Financial award simulation based on historical Code4rena payouts.

2. Patch Mode:

   • Task: The agent modifies the vulnerable codebase to fix the identified issues.
   • Constraints: The agent has access to the test suite and dependencies. It must not break existing functionality (tests must pass) and must prevent the exploit.
   • Grading: The system applies the agent's changes, runs the original test suite (excluding tests that relied on the bug), and attempts to run unseen exploit scripts. A patch passes if tests pass and exploits fail.

3. Exploit Mode:

   • Task: The agent is given a funded wallet, an RPC endpoint to a local Ethereum instance, and the contract addresses. It must autonomously analyze the chain, construct transactions, and execute an end-to-end exploit to drain funds.
   • Environment: Agents run in isolated Docker containers. The evaluation uses a Rust-based re-execution framework (ploit) that replays agent transactions on a local Anvil chain to verify state changes.
   • Grading: Custom scripts analyze the final blockchain state (balances and events). Success is binary: did the agent successfully drain the target funds?

C. Infrastructure & Security

• Anti-Cheating: The framework uses a veto proxy to block non-standard JSON-RPC methods (e.g., anvil_impersonateAccount) that could allow agents to manipulate state directly rather than exploiting logic.
• Determinism: The local blockchain is forked/recreated with fixed parameters to ensure reproducible grading.
• Comprehensive Coverage: Agents are evaluated on finding all vulnerabilities in a repo, not just the first one, to prevent "lucky" single-success scoring.

3. Key Contributions

1. First End-to-End Smart Contract Benchmark: EVMbench is the first framework to evaluate the full cycle of discovery, remediation, and exploitation in a realistic, programmatic environment.
2. Programmatic Grading: Unlike previous benchmarks relying on human review, Patch and Exploit modes use automated, deterministic grading based on on-chain state and test suites.
3. Realistic Attack Surface: The framework simulates a "dark forest" environment where agents must reason about economic incentives, stateful interactions, and adversarial actors (simulated via the grading logic).
4. Open Source: The authors released the dataset, evaluation harness, and tooling to facilitate future research in AI security.

4. Results

The authors evaluated frontier models (OpenAI o3, GPT-5 variants, Claude Opus 4.5/4.6, Gemini 3 Pro) across all three modes.

• Performance Overview:
  • Exploit Mode: GPT-5.3-Codex achieved the highest score (71.0%), demonstrating the ability to execute complex, multi-step exploits (e.g., flash loan attacks) end-to-end.
  • Patch Mode: GPT-5.3-Codex also led here with 41.7%, though scores were generally lower than in Exploit, indicating that fixing bugs is harder than exploiting them in some contexts, or that discovery is the bottleneck.
  • Detect Mode: Claude Opus 4.6 scored highest (45.9%), followed closely by GPT-5.3-Codex.
• Key Findings:
  • Discovery is the Bottleneck: Providing "hints" (e.g., pointing to specific files or mechanisms) dramatically improved Patch and Exploit scores (e.g., GPT-5.2 Exploit score jumped from ~39% to ~78% with medium hints). This suggests models possess the reasoning skills to fix/exploit once the vulnerability is located.
  • Coverage vs. Isolated Wins: Models often found one vulnerability but missed others in the same codebase. High scores in one mode did not guarantee success in finding all issues.
  • Scaffolding Matters: Models run with specialized coding scaffolds (e.g., Codex CLI) significantly outperformed those using generic agents (e.g., OpenCode), highlighting the importance of tooling.
  • Economic Impact: The best agents could theoretically drain significant funds. The "Detect" awards (simulated) showed that top models could earn tens of thousands of dollars in simulated bounties, though still far below the theoretical maximum.

5. Significance and Implications

• Dual-Use Risk: The results confirm that advanced AI agents can autonomously discover and exploit smart contract vulnerabilities, posing a direct financial risk to the crypto ecosystem. The ability to execute end-to-end exploits against live instances is a critical safety signal.
• Defensive Utility: Conversely, these capabilities can be harnessed for defense. AI agents can serve as automated auditors, potentially finding bugs faster and more comprehensively than human teams, provided they are guided correctly.
• Evaluation Standard: EVMbench sets a new standard for measuring AI safety in economically critical domains. It moves beyond "can the model write code?" to "can the model break and fix complex financial systems?"
• Future Directions: The authors note limitations, such as the lack of cross-chain support and mempool reasoning (MEV), and suggest expanding to other chains (Solana) and cryptographic primitives (ZK circuits) as the next steps.

In conclusion, EVMbench demonstrates that frontier AI agents are already capable of navigating the "dark forest" of blockchain security, capable of both draining funds and patching vulnerabilities, necessitating rigorous monitoring and defensive deployment strategies.