💻 computer science

A Human-in-the-Loop Framework for Detecting Logical Vulnerabilities in Smart Contracts

This paper proposes a Human-in-the-Loop framework that combines automated analysis with targeted human review to detect logical vulnerabilities in smart contracts, demonstrating significant improvements in precision and F1-score while reducing false positives on the SmartBugs Curated dataset.

Original author: Alireza Assadzadeh

Published 2026-11-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original author: Alireza Assadzadeh

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the author. For technical accuracy, refer to the original paper. Read full disclaimer

In the digital world, a blockchain is a shared record book that many computers keep in sync, ensuring that no single person can alter the past. At the heart of this system lies a consensus protocol, a set of rules that allows these separate computers to agree on what is true without needing a central boss. On top of this foundation, developers write smart contracts, which are self-executing programs that automatically carry out agreements, such as transferring money or managing assets, once specific conditions are met. While these technologies promise a new era of trust and automation, they are not immune to failure. Just as a physical lock can be picked or a building can have a structural flaw, the code behind these systems can contain hidden errors. Some of these errors are obvious mistakes in the code, but the most dangerous ones are logical vulnerabilities. These are not typos or broken syntax; they are flaws in the actual plan or business logic, where the code does exactly what it was told to do, but the instructions themselves were wrong. Because these errors depend heavily on the specific purpose of the program, they are notoriously difficult for computers to spot on their own.

A research team led by Alireza Assadzadeh at Islamic Azad University Tabriz Branch has proposed a way to solve this specific problem by bringing a human back into the process. The researchers argue that while automated tools are excellent at scanning code for known patterns, they struggle with the subtle, context-dependent errors that define logical vulnerabilities. To address this, they designed a framework called Human-in-the-Loop, which acts as a partnership between a fast, automated scanner and a careful human expert. The system works in stages. First, an automated tool scans a smart contract and flags potential issues, assigning each warning a confidence score that indicates how sure the tool is about the problem. Instead of showing every single warning to a human, which would be overwhelming, the system filters the list. It only passes the warnings with a high confidence score to the human expert for a final look. The expert then reviews these specific cases, deciding whether they are genuine threats or false alarms. This approach aims to keep the speed of machines while adding the nuance of human judgment.

To test if this method actually works, the researchers ran experiments using a collection of 143 smart contracts that were already known to contain various types of security flaws. They simulated the behavior of a standard automated scanner and then applied their new filtering and review process. The results showed a clear improvement in accuracy. When relying solely on the automated tool, the system correctly identified real problems but also raised many false alarms, leading to a precision score of roughly 0.71. Precision, in this context, measures how many of the flagged issues were actually real problems. After introducing the human review step, the precision jumped to roughly 0.84. This means that the number of false alarms dropped by more than half, specifically by 52.2 percent, while the system did not miss a single real vulnerability that the machine had originally found. The overall measure of the system's effectiveness, known as the F1-score, also rose from roughly 0.83 to 0.91.

The study suggests that this hybrid approach offers a practical solution to a persistent problem in blockchain security. By letting a human expert handle only the most likely candidates, the system reduces the workload while significantly increasing trust in the results. The researchers note that this is particularly important for logical vulnerabilities, which often require an understanding of the intended business rules rather than just a check of the code's structure. While the findings are based on a controlled simulation using existing data rather than a live deployment on a major network, the results indicate that integrating human expertise can meaningfully improve the reliability of detecting these complex errors. The work points toward a future where securing decentralized applications involves not just better software, but a structured collaboration between automated tools and human insight.

Technical Summary: A Human-in-the-Loop Framework for Detecting Logical Vulnerabilities in Smart Contracts

Problem Statement
Blockchain systems face a spectrum of security challenges ranging from the consensus layer to the application layer. While consensus protocols (e.g., Proof-of-Work, Proof-of-Stake) have been extensively studied regarding their trade-offs in security, scalability, and decentralization, significant gaps remain in securing the application layer, specifically smart contracts. Although automated tools (static analysis, symbolic execution, fuzzing, and machine learning) have improved the detection of well-known syntactic bugs like reentrancy and integer overflows, they struggle with logical vulnerabilities. These vulnerabilities arise from flawed business logic, incorrect assumptions, or incomplete access control. Because logical flaws are highly context-dependent and require an understanding of the contract's intended behavior, purely automated detectors often produce high rates of false positives or fail to identify subtle errors that lead to substantial financial losses. Furthermore, existing literature often treats consensus mechanisms and smart contract vulnerabilities in isolation, lacking a unified perspective that incorporates human expertise into the detection process.

Methodology
The paper proposes a Human-in-the-Loop (HITL) framework designed to bridge the gap between automated speed and human contextual judgment. The methodology consists of a three-stage pipeline:

Automated Detection: A static analysis tool scans smart contract source code to identify potential vulnerabilities. Each reported issue is assigned a confidence score.
Confidence-Based Filtering: To manage the cognitive load on human experts, a threshold ( $\theta$ ) is applied. In the proposed experiments, $\theta = 0.55$ . Only warnings with a confidence score equal to or exceeding this threshold are forwarded for human review. This step filters out low-confidence noise while retaining high-risk findings.
Human Review: A human expert examines the filtered warnings to determine if they are true positives or false positives. The expert validates the findings against the contract's business logic and access control policies, rejecting incorrect warnings that automated tools cannot reliably distinguish.

Formally, if $M(c)$ is the set of issues reported by the tool for contract $c$ , the subset sent to the human is $M_\theta(c) = \{ v \in M(c) \mid \text{confidence}(v) \geq \theta \}$ . The final detection result $D(c)$ is the output of the human review function $H(\cdot)$ applied to this subset.

Key Contributions
The paper makes the following specific contributions:

Structured Overview: It provides a comprehensive analysis of security issues across major consensus protocols, highlighting their strengths, weaknesses, and open challenges.
Vulnerability Analysis: It analyzes logical vulnerabilities in smart contracts, emphasizing the specific limitations of purely automated detection methods regarding context-dependent flaws.
HITL Review: It reviews existing Human-in-the-Loop approaches in blockchain security, identifying current capabilities and gaps in the literature.
Proposed Framework & Evaluation: It proposes and empirically evaluates a simple HITL framework using the SmartBugs Curated dataset (143 annotated Solidity contracts). The evaluation demonstrates that integrating human review significantly reduces false positives while maintaining detection coverage.
Unified Perspective: It outlines a unified analytical perspective connecting consensus-level threats to logical vulnerabilities and HITL-based mitigation strategies.

Experimental Results
The framework was evaluated on the SmartBugs Curated dataset, which includes various vulnerability types such as reentrancy, access control, and arithmetic issues. The study compared an "Automated Only" setting against the "HITL" setting.

Precision: Improved from 0.7125 (Automated) to 0.8382 (HITL).
F1-Score: Improved from 0.8321 (Automated) to 0.9120 (HITL).
False Positives (FP): Reduced by 52.2% (from 23 to 11).
Recall: Remained constant at 1.0000, indicating that the human review process successfully filtered out incorrect warnings without missing any true vulnerabilities flagged by the automated tool.

The results indicate that the confidence-based filtering effectively reduced the workload for human experts while the human review stage successfully eliminated a substantial portion of false positives.

Significance and Claims
The paper claims that integrating human expertise into the detection pipeline is a practical and effective strategy for improving the reliability of logical vulnerability detection. The author argues that while automated tools are essential for scalability, they cannot yet reliably handle the semantic understanding required for complex business logic. By combining automated analysis with targeted human review, the framework achieves a meaningful reduction in false positives without sacrificing recall.

The paper maintains a modest stance, noting that the results are based on a controlled simulation reflecting typical tool behaviors reported in prior empirical studies. It acknowledges that a full manual review of all contracts would provide stronger evidence but asserts that the current findings demonstrate the practical value of the HITL approach. The work concludes that securing blockchain systems requires attention to both technical mechanisms and human factors, suggesting that future research should focus on advanced confidence estimation, larger datasets for logical vulnerabilities, and interactive visualization tools to further reduce cognitive load.

Technical Summary: A Human-in-the-Loop Framework for Detecting Logical Vulnerabilities in Smart Contracts

More like this