Beyond F1: A Study of LLM Reliability in Smart Contract Vulnerability Detection
This study evaluates 14 large language models on smart contract vulnerability detection, revealing that model selection is more critical than prompting strategies or increased inference-time computation, as larger frontier models demonstrate superior reliability while smaller models and extended reasoning techniques can degrade performance.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the digital world of decentralized finance, money moves through self-executing programs called smart contracts. These are like digital vending machines that automatically dispense assets when specific conditions are met, operating without human managers or banks. Because these programs run on a blockchain, once they are deployed, they are effectively permanent; if a flaw is found after the machine is built, it is often impossible to fix. This immutability makes security critical, as even a tiny mistake can lead to the loss of billions of dollars. For years, researchers have relied on automated tools to scan these contracts for errors, but these tools often struggle to understand the context of the code, frequently raising false alarms or missing subtle dangers. Recently, a new type of artificial intelligence known as a large language model has emerged, capable of reading and understanding code much like a human programmer. This has raised a hopeful question: can these intelligent systems replace or improve upon traditional security checks to find vulnerabilities before they cause harm?
A recent study set out to answer this by putting fourteen different artificial intelligence models to the test. The researchers gathered fifty-four real-world smart contracts, including some known to have security flaws and others that were perfectly clean, to see how well the models could distinguish between the two. They did not simply ask the models to look at the code; they tried four different ways of asking, ranging from a simple direct question to complex, step-by-step instructions that forced the models to think through their analysis. They also tested whether giving the models more time to "think" during the process improved their results, a feature some modern models offer to reason through problems more deeply. The goal was to determine if these powerful tools could reliably find bugs without mistakenly accusing safe code of being dangerous.
The results revealed a sharp divide in performance that challenges some common assumptions about how these models work. The study found that simply giving a model more time to think or asking it to follow a detailed reasoning process did not always make it better. In fact, for one of the top-performing models, enabling this extra thinking mode actually made it worse, causing it to miss genuine vulnerabilities and lowering its overall accuracy. This suggests that more computational effort does not automatically translate to better security analysis. Instead, the most important factor was simply which model was chosen. The best models, which are large and commercially developed, proved to be highly reliable, correctly identifying vulnerabilities while rarely making mistakes on safe code.
In stark contrast, the smaller, open-source models tested in the study behaved very differently. While these smaller models were excellent at spotting potential problems, they suffered from a critical flaw: they were unable to tell the difference between a vulnerable contract and a safe one. They flagged every single clean contract as dangerous, effectively screaming "wolf" at every turn. The researchers coined a specific term for this tendency to hallucinate threats where none exist, calling it a "Basilisk Rate." A model with a high rate is useless for security because it would overwhelm engineers with false alarms. The study showed that the six best-performing models had a perfect record of never flagging clean code, while the three smallest models flagged every clean contract as vulnerable.
The investigation also highlighted a specific threshold in the size of these models that seems to determine their reliability. The models that performed well generally had a larger number of internal parameters, which can be thought of as the complexity of their internal knowledge base. The study suggests that a transition point exists around a specific size, where models below that threshold tend to hallucinate constantly, while those above it become precise and trustworthy. Interestingly, one large open-source model managed to bridge this gap, performing nearly as well as the top commercial models, but the smaller ones remained unreliable.
Ultimately, the study concludes that for the specific task of finding security flaws in smart contracts, the choice of the artificial intelligence model matters far more than how it is prompted or how much time it is given to think. While these advanced tools show great promise, they are not a magic solution that works the same way for every system. The most effective approach involves selecting a model that has been proven to be precise and avoiding those that are prone to seeing problems that do not exist. For the future of digital finance, this means that security teams must carefully vet which artificial intelligence tools they use, ensuring they rely on systems that can distinguish between a real threat and a harmless piece of code.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.