Technical Summary: An Empirical Analysis of Vulnerability Detection Tools for Solidity Smart Contracts

Problem Statement

The rapid adoption of blockchain technology has elevated the importance of securing smart contracts (SCs), which automate critical business logic. Despite the proliferation of automated vulnerability detection tools, empirical evidence suggests significant reliability issues, including high false positive (FP) rates and inconsistent detection capabilities. Existing evaluations often rely on datasets that are either automatically labeled (inheriting tool biases), limited in scale, or annotated only at the contract or function level. Furthermore, there is a critical gap in understanding how well these tools perform on real-world, complex contracts compared to curated, simplified benchmarks. Additionally, the emergence of Large Language Models (LLMs) as detection tools requires rigorous evaluation to determine their generalizability beyond controlled environments.

Methodology

The authors conducted a comprehensive empirical study involving the following key methodological steps:

1. Dataset Construction:

   • The study constructed a manually annotated dataset comprising 2,182 Solidity smart contract instances.
   • Vulnerabilities were labeled at the line-of-code level, addressing the granularity preference identified in a motivating survey of developers and researchers.
   • Data sources included:
     • SmartBugs Results: A stratified sample of 1,501 contracts extracted from a larger corpus of 47,587, selected based on tool outputs (requiring at least two tools to flag a vulnerability).
     • SmartBugs Curated: 142 manually annotated contracts (139 confirmed as vulnerable).
     • ZEUS Dataset: 538 contracts retrieved via Etherscan API from the Kalra et al. (2018) dataset, re-evaluated manually.
   • A third-party evaluator double-checked non-vulnerable labels, and discrepancies were resolved through consensus.

2. Tool Evaluation:

   • Traditional Tools: The study evaluated 19 vulnerability detection tools from the SmartBugs 2.0 framework (excluding HoneyBadger, which targets honeypots). These included static analyzers (e.g., Slither, Solhint, Securify), dynamic analyzers/fuzzers (e.g., Manticore, Sfuzz), and symbolic execution tools (e.g., Osiris, Oyente).
   • LLM Evaluation: ChatGPT-4o was evaluated as a line-level vulnerability detector. The model was prompted to identify vulnerabilities based on the DASP TOP 10 taxonomy.
   • Scenarios: Evaluations were conducted under two scenarios regarding arithmetic vulnerabilities: one including them (relevant for Solidity < 0.8.0) and one excluding them (accounting for default overflow checks in Solidity >= 0.8.0).

3. LLM Specifics:

   • ChatGPT-4o was tested on the SmartBugs Curated dataset and a random sample of 400 real-world contracts from the SmartBugs Results dataset.
   • The study analyzed the impact of contract complexity (measured by cyclomatic complexity and lines of code) and potential data leakage (training data overlap) on LLM performance.

4. Combination Strategy:

   • Tools were clustered using the k-means algorithm based on their detection behavior.
   • Representative tools from each cluster were selected to form a complementary detection suite.

Key Contributions

1. Dataset Release: The release of the largest manually analyzed dataset of smart contracts to date, featuring line-level vulnerability annotations for 2,182 instances. This dataset addresses the lack of fine-grained ground truth in existing benchmarks.
2. Extensive Tool Evaluation: The most extensive empirical evaluation of 19 state-of-the-art tools (plus an LLM approach) mapped to the DASP TOP 10 taxonomy, evaluated against a manually verified ground truth.
3. LLM Performance Analysis: A demonstration that LLM-based detection (ChatGPT-4o) exhibits significantly lower effectiveness on real-world contracts compared to curated benchmarks, highlighting issues with generalization and potential data leakage.
4. Optimal Tool Combination: Identification of a specific combination of three tools (Conkas, Slither, and Smartcheck) that, when used together, detect up to 76.78% of manually annotated vulnerabilities with an average execution time of less than one minute.

Results

Tool Performance (RQ1)

• No Single Tool is Sufficient: No single tool detected all vulnerability classes in the DASP TOP 10 taxonomy.
• Specialization:
  • Osiris was most effective for arithmetic vulnerabilities.
  • Smartcheck excelled at Denial of Service (DoS) detection.
  • Solhint showed strong capabilities for Access Control, Unchecked Low-Level Calls, and Bad Randomness, though it suffered from a high false positive rate.
  • Conkas and Slither emerged as the most effective tools overall, with Conkas performing better when arithmetic vulnerabilities were included and Slither performing better when they were excluded (Solidity >= 0.8.0).
• Execution Time: Significant trade-offs exist between recall and execution time. For instance, Slither achieved high recall (0.68) with a very low average execution time (1.14s), whereas Sfuzz had similar recall but required over 582 seconds on average.
• Undetected Classes: Tools struggled significantly with Front Running, Short Addresses, and Bad Randomness, though some improvements were noted compared to prior studies (e.g., Securify detecting Front Running).

LLM Effectiveness (RQ2)

• Curated vs. Real-World: ChatGPT-4o achieved moderate recall (47.67%) and higher precision on the SmartBugs Curated dataset compared to previous reports. However, on 400 real-world contracts, performance collapsed (Precision: 5.6%, Recall: 3.3%, F1: 1.9%).
• Complexity Factor: Real-world contracts were significantly larger and more complex (higher cyclomatic complexity) than curated benchmarks. The study suggests that LLM effectiveness degrades as contract size and complexity increase.
• Data Leakage: The study hypothesizes that the high performance on curated datasets may be partly due to data leakage, as these contracts often predate the model's training cutoff or are present in public repositories used for training.

Tool Combination (RQ3)

• By clustering tools and selecting complementary representatives, the authors identified a trio (Conkas, Slither, Smartcheck) that collectively detected 73.38% of all vulnerabilities (including arithmetic) and 76.78% of non-arithmetic vulnerabilities.
• This combination was statistically superior to individual tools and previous combinations (Fisher's Exact Test, p < 0.001), offering a high coverage rate with minimal computational overhead (average < 1 minute).

Significance and Claims

The paper claims to make a significant contribution to the field of smart contract security by:

1. Providing a Reliable Benchmark: The manually annotated, line-level dataset serves as a robust ground truth for future tool evaluation, moving beyond the biases of automatically labeled datasets.
2. Highlighting Evaluation Gaps: The study underscores the discrepancy between tool performance on simplified benchmarks versus real-world deployments, warning against overestimating tool efficacy based solely on curated datasets.
3. Demonstrating LLM Limitations: It provides empirical evidence that general-purpose LLMs, while promising, currently lack the reliability required for real-world smart contract auditing without further domain-specific fine-tuning or validation.
4. Offering a Practical Solution: The identification of a complementary tool combination offers a practical, scalable strategy for developers and auditors to maximize vulnerability detection while managing computational costs.

The authors conclude that while tools have improved, significant challenges remain, particularly regarding false positives, the detection of underrepresented vulnerability classes, and the generalization of AI-based methods to complex, real-world scenarios.