Reentrancy Detection in the Age of LLMs

1. Problem Statement

Reentrancy remains the most critical vulnerability class in Ethereum smart contracts, responsible for approximately $908.6 million in losses. However, the ecosystem of detection tools and datasets is increasingly unreliable due to:

• Tool Obsolescence: Many static analyzers and formal verification tools rely on outdated compiler versions (pre-Solidity 0.8) and obsolete definitions of reentrancy (strictly adhering to the "Checks-Effects-Interactions" or CEI pattern). They often crash or fail on modern syntax, error handling, and advanced language features.
• Inconsistent Ground Truth: Existing datasets used for training Machine Learning (ML) models are often labeled by the very tools being evaluated. This creates a circular dependency where tools inherit each other's biases, false positives, and outdated assumptions.
• Lack of Robustness: Current detectors struggle with semantic corner cases, such as read-only reentrancy, proxy-mediated flows, and complex modifier-based guards, leading to high rates of disagreement between tools.

2. Methodology

The authors adopted a dependability-oriented perspective, distinguishing between reliability (correctness of predictions) and robustness (ability to function across evolving language versions).

A. Dataset Construction

To break the cycle of tool-induced bias, the authors constructed two new, manually verified datasets:

1. Aggregated Benchmark: A consolidation of 432 real-world contracts from three prior academic datasets (CGT, HuangGai, and a large-scale study).
   • Process: Contracts were deduplicated, filtered for compilability, and manually re-labeled by three experts using a unified operational definition.
   • LLM Assistance: LLMs were used as a "second opinion" to flag edge cases and explain ambiguous scenarios, leading to the correction of mislabeled contracts.
   • Composition: 122 reentrant and 314 safe contracts.
2. Reentrancy Scenarios Dataset (RSD): A suite of 143 handcrafted Minimal Working Examples (MWEs) designed to stress-test specific semantic patterns.
   • Focus: Modern Solidity 0.8+ features, including delegatecall, staticcall, cross-function reentrancy, ERC20 logic, and various guard mechanisms (mutexes, modifiers).
   • Variants: Each scenario includes syntactic variations (e.g., folding, error handling, inheritance) to test robustness against structural changes.

B. Unified Labeling Procedure

The authors defined a three-step operational procedure for labeling, which serves as the ground truth:

1. Unfolding: Semantically preserving transformation of the code (expanding local functions, modifiers, and inheritance) to flatten the control flow.
2. Syntactic Detection: Identifying external calls followed by state updates (side effects).
3. Semantic Comparison: Comparing the contract state resulting from nested reentrant calls against the state of sequential, non-reentrant calls. If the final states diverge, the contract is vulnerable.

C. Experimental Evaluation

The authors evaluated 31 systems across three categories:

• 12 Formal/Static Analysis Tools: (e.g., Mythril, Slither, Securify, Oyente+, Aderyn).
• 10 Machine Learning Models: Including traditional ML (Random Forest, SVM) and Deep Learning (CodeBERT, BiLSTM).
• 9 Large Language Models (LLMs): Both commercial (GPT-4o, GPT-4.1, o3-mini, Gemini) and open-source (Qwen, Phi, GPT-OSS), evaluated in a zero-shot setting.

3. Key Contributions

1. Dependability Assessment: The first systematic evaluation revealing that the reentrancy detection ecosystem is "fractured," with many tools failing entirely on modern code or producing inconsistent results.
2. New Benchmarks: Release of the Aggregated Benchmark (high-confidence real-world contracts) and the RSD (semantic stress tests), both manually verified to eliminate tool-induced noise.
3. LLM Superiority: Demonstration that general-purpose LLMs outperform specialized static analyzers and ML models in both accuracy and robustness, without requiring maintenance or retraining.
4. Human-in-the-Loop Utility: Evidence that LLM-generated explanations assist human auditors in identifying mislabeled cases and clarifying ambiguous vulnerabilities, improving dataset quality.

4. Key Results

Performance Metrics (F1 Score & Accuracy)

• Aggregated Benchmark:
  • Best Static Tool: Slither (F1: 0.92).
  • Best ML Model: FFNN (F1: 0.86).
  • Best LLM: o3-mini (F1: 0.96).
• Reentrancy Scenarios Dataset (RSD):
  • Best Static Tool: Slither (F1: 0.76).
  • Best ML Model: XGBoost (F1: 0.72).
  • Best LLM: gpt-oss (F1: 0.85).

Robustness and Consistency

• Error Rates: Static tools exhibited high error rates on the RSD (e.g., Vandal: 78%, Conkas: 100% failure) due to unsupported Solidity versions. LLMs had 0% error rates, successfully analyzing all inputs.
• Tool Agreement: On the Aggregated Benchmark, only 38 out of 122 reentrant contracts were agreed upon by at least 6 different tools. This indicates a severe lack of consensus even on "known" vulnerabilities.
• Maintenance: Static tools require constant rule updates to support new Solidity features. LLMs demonstrated inherent adaptability to language evolution without manual intervention.

Qualitative Findings

• Explanations: LLMs provided concise, readable explanations that helped human experts identify 9 mislabeled contracts in the Aggregated Benchmark. In contrast, static tool logs were often cryptic or required significant manual effort to interpret.
• Semantic Reasoning: LLMs successfully identified complex patterns (e.g., read-only reentrancy via staticcall) that rule-based tools missed because they relied on syntactic patterns rather than semantic understanding.

5. Significance and Implications

• Paradigm Shift: The paper argues that static analysis tools are no longer a reliable "ground truth" for smart contract security pipelines due to their fragility against language evolution.
• LLM Viability: LLMs offer a robust, low-maintenance alternative that outperforms traditional methods. While they lack formal guarantees, their semantic reasoning capabilities make them superior for detecting subtle, modern vulnerabilities.
• Future Direction: The authors advocate for hybrid approaches that combine the formal guarantees of static analysis with the adaptability of LLMs. They also emphasize the need for high-quality, human-verified datasets rather than those generated by legacy tools.
• Practical Impact: Security auditors and developers should reconsider relying solely on static analyzers for modern Solidity code and consider integrating LLMs (with human oversight) into their verification workflows.