Re-Evaluating EVMBench: Are AI Agents Ready for Smart Contract Security?

This paper challenges the optimistic narrative of fully automated AI auditing by demonstrating that EVMbench's results are compromised by data contamination and narrow evaluation, revealing that current AI agents lack the stability and end-to-end exploitation capabilities to replace human auditors in smart contract security.

The Gist

Imagine the world of cryptocurrency smart contracts as a massive, high-stakes bank vault. Before anyone can put their money inside, they need to check the locks, the alarms, and the steel doors for any weaknesses. This is called "auditing."

Recently, a group of tech giants released a report (called EVMbench) claiming that AI agents (super-smart computer programs) were ready to take over this job entirely. They said, "AI can find 45% of the bugs and actually break into 72% of the test vaults! We don't need humans anymore!"

This paper, written by researchers from Zhejiang University and BlockSec, is like a skeptical detective saying, "Hold on a minute. Let's check the math."

Here is the breakdown of what they found, using simple analogies:

1. The "Practice Exam" Problem (Data Contamination)

The Old Report: The original AI test used questions from a "practice exam" (Code4rena contests) that the AI models might have already seen in their training data.
The New Reality: It's like giving a student a math test where the answers were printed in the back of their textbook. Of course, they got a high score!
The Fix: The researchers created a brand new test using 22 real-world bank robberies that happened after the AI models were built. The AI had never seen these crimes before.
The Result: When the AI faced these "fresh" crimes, its performance dropped significantly. It could spot the idea of a crime sometimes, but it couldn't actually pull off the heist.

2. The "Toolbox" Matters More Than the "Brain"

The Old Report: The original test mostly paired each AI with its own brand's tools (e.g., Google's AI with Google's tools, Anthropic's AI with Anthropic's tools).
The New Reality: The researchers found that the tools (the "scaffold") matter just as much as the brain (the AI model).
The Analogy: Imagine a master chef (the AI). If you give them a rusty, broken knife (a bad tool), they can't cook a great meal. But if you give them a sharp, professional knife (a good open-source tool), they suddenly become a genius.
The Result: An older, open-source toolset actually helped the AI perform better than the fancy, brand-new tools the original report used. The original report blamed the "brain" for low scores, but it was actually the "knife" that was dull.

3. Finding the Bug vs. Stealing the Money

The Old Report: The original conclusion was: "Finding the bug is the hard part. Once found, stealing the money is easy."
The New Reality: The researchers found that finding the bug and stealing the money are two completely different skills.
The Analogy: Imagine a detective who can look at a house and say, "Hey, the back door is unlocked!" (Finding the bug). That's great. But can that same detective then sneak in, avoid the laser grid, pick the safe, and walk out with the diamonds without getting caught? (Exploiting the bug).
The Result: The AI was okay at pointing out the unlocked door (finding bugs), but it completely failed at actually stealing the diamonds in real-world scenarios. In the new test, the AI tried to steal from 110 real-world scenarios and succeeded 0 times.

4. The "Rankings" Are Unstable

The Old Report: They said, "Model A is the best, Model B is second."
The New Reality: The rankings changed depending on which tools you used or which specific test you gave them.
The Analogy: It's like saying "Usain Bolt is the fastest runner in the world." But then you test him on a track made of mud, and suddenly a marathon runner wins. If you test him on a treadmill, a different runner wins. You can't declare a single "champion" when the rules and the track keep changing.

The Bottom Line: What Should We Do?

The paper concludes that AI is not ready to replace human auditors entirely. It's not a "magic wand" that solves everything.

• For Developers: Think of AI as a spell-checker. It's great at catching typos (common, well-known bugs like missing passwords). But it can't write the whole story for you, and it might miss the deep, complex plot holes.
• For Security Firms: The best approach is a "Human-in-the-Loop" workflow.
  • The AI acts as a tireless intern who scans thousands of pages of code quickly to find the obvious mistakes.
  • The Human acts as the senior detective who takes the AI's notes, adds context, understands the complex logic, and makes the final judgment.

In short: AI is a powerful assistant, but it's not the boss yet. If you let it work alone, you might think your vault is secure when it's actually wide open. The future isn't "AI vs. Humans"; it's "AI + Humans" working together.

Technical Summary

Here is a detailed technical summary of the paper "Re-Evaluating EVMBench: Are AI Agents Ready for Smart Contract Security?"

1. Problem Statement

The paper addresses the validity of EVMbench, a benchmark released by OpenAI, Paradigm, and OtterSec, which claimed that AI agents are nearing the capability for fully automated smart contract auditing. EVMbench reported that agents could detect up to 45.6% of vulnerabilities and exploit 72.2% of a curated subset, leading to the conclusion that "discovery" is the primary bottleneck, not exploitation.

The authors identify two critical flaws in EVMbench's experimental design that invalidate these optimistic conclusions:

1. Narrow and Confounded Evaluation Scope: EVMbench tested only 14 configurations, mostly pairing models with their native vendor scaffolds (e.g., Claude with Claude Code). This confounds model capability with scaffold performance, failing to isolate whether results are due to the model or the tooling.
2. Data Contamination: The benchmark relies on Code4rena audit reports published before the release of many evaluated models (late 2025/2026). This raises the possibility that high scores reflect memorization of training data rather than genuine reasoning capabilities. Furthermore, curated contest data may not reflect the complexity of real-world production exploits.

2. Methodology

The authors conducted a rigorous re-evaluation using an expanded scope and a contamination-free dataset.

• Expanded Agent Configurations:
  • Scale: Increased from 14 to 26 configurations.
  • Models: Covered four model families (Claude, GPT, Gemini, GLM) including current-generation models (e.g., Gemini 3.1 Pro, GPT-5.3-Codex).
  • Scaffolds: Systematically varied scaffolds across three types: Vendor-native (Claude Code, Codex CLI) and Open-source (OpenCode). This allowed for the isolation of "scaffold effects" from "model effects."
• New Dataset (Incidents):
  • Constructed a contamination-free dataset of 22 real-world security incidents.
  • Criteria: All incidents occurred after the release dates of every evaluated model (post-mid-February 2026), ensuring they were outside the models' training windows.
  • Nature: Unlike curated contest data, these involved production-deployed code, confirmed financial losses, and required agents to fetch on-chain state and execute profitable transactions without pre-labeled hints.
• Evaluation Tasks:
  • Detect: Identifying vulnerabilities (graded by model-based judges).
  • Exploit: Executing end-to-end attacks on forked chain snapshots to drain funds (stricter than EVMbench's setting).
  • Note: The "Patch" task was excluded as EVMbench data suggested its difficulty is largely reducible to detection difficulty.
• Grader Reliability: Validated the model-based grading system using three different judge models, confirming 99.2% accuracy and resistance to prompt injection.

3. Key Contributions

1. Demonstration of Instability: Proved that AI agent rankings are highly unstable. A model's performance varies significantly based on the scaffold used, the reasoning effort level, the specific task (detection vs. exploitation), and the dataset (curated vs. real-world).
2. Scaffold Impact: Showed that the choice of scaffold (tooling) materially affects results. An open-source scaffold (OpenCode) outperformed vendor alternatives by up to 5 percentage points in controlled comparisons, a variable EVMbench failed to control.
3. Real-World Capability Gap: Directly contradicted EVMbench's conclusion that "discovery is the bottleneck." While agents detected up to 65% of real-world vulnerabilities, 0% of the 110 agent-incident pairs resulted in a successful end-to-end exploitation.
4. Contamination-Free Benchmarking: Introduced a methodology for testing agents on strictly post-release data to eliminate memorization bias.

4. Key Results

A. Detection Performance

• Curated Data (EVMbench): The best agent (Claude Opus 4.6) achieved 47.5% detection. Rankings shifted drastically when scaffolds or model versions changed (e.g., Gemini 3.1 Pro jumped from 16.7% to 35.0% with a version update).
• Real-World Data (Incidents): The best agent detected 65% of vulnerabilities. However, performance was inconsistent; Gemini 3.1 Pro dropped from 2nd place on EVMbench to last place (30.0%) on the Incidents dataset.
• Pattern Recognition: Agents reliably detected well-known patterns (missing access controls, reentrancy) but failed on complex, protocol-specific logic (e.g., cross-chain replay attacks, nested signature state machines).

B. Exploitation Performance

• Curated Data: Agents achieved up to 61.1% success on EVMbench tasks.
• Real-World Data: 0% success rate. No agent successfully executed a profitable end-to-end exploit on any of the 22 real-world incidents.
• Failure Modes: Agents struggled with multi-step protocol interactions, understanding external dependencies, and chaining token approvals/flash loans required for profitable attacks. They often got stuck reading code without converging on an attack strategy.

C. Scaffold and Reasoning Effects

• Scaffold: OpenCode outperformed vendor scaffolds in 5 of 6 controlled comparisons.
• Reasoning Effort: Increasing reasoning tokens did not always help. For GPT-5.2, low reasoning effort outperformed high effort by 8 percentage points on exploitation, suggesting "overthinking" can degrade performance.

5. Significance and Implications

The paper challenges the narrative that fully automated AI auditing is imminent.

• For Developers: Agent scans are useful as a pre-deployment check for common, well-known vulnerabilities (catching ~65% of real-world incidents in the study). However, relying solely on them creates a false sense of security, as >30% of vulnerabilities (and complex logic flaws) remain undetected.
• For Audit Firms: The optimal path is a Human-in-the-Loop (HITL) Agentic Workflow.
  • AI Role: Handle "breadth" by scanning large codebases for common patterns and triaging easy issues.
  • Human Role: Provide "depth" via protocol-specific knowledge, adversarial reasoning, and filtering false positives.
  • Evidence: EVMbench's own hint experiments showed that when agents received human context, exploit scores jumped from ~62% to 95.8%.
• For Research: Future benchmarks must control for scaffold variables, use contamination-free datasets, and measure precision (false positives) alongside recall. Current metrics overstate practical utility by ignoring the noise generated by false alarms.

Conclusion: AI agents possess real but bounded capabilities. They are effective tools for augmenting human auditors but are not yet ready to replace human judgment in smart contract security.