Pre-Model Representation Failures in GNN-Based Smart Contract Vulnerability Detection

In the digital world of decentralized finance, money moves through self-executing computer programs called smart contracts. These programs live on a blockchain, a public ledger that records every transaction, and they are designed to handle real assets without human intermediaries. Because these contracts manage valuable funds, a single hidden flaw can lead to immediate and irreversible theft. To protect against this, researchers have developed automated systems that scan code to find weaknesses before they are exploited. One popular approach uses a type of artificial intelligence known as a graph neural network. To understand how this works, imagine taking a complex piece of code and turning it into a map of connected dots and lines, where the dots represent parts of the program and the lines show how they interact. The computer then studies this map to decide if the program is safe or dangerous. The hope is that by looking at the structure of the code rather than just reading it line by line, these systems can spot subtle traps that humans might miss.

A team of researchers from Carnegie Mellon University Africa decided to test the very foundation of this approach. They focused on a specific system called GNNSCVulDetector, which is widely used to find a dangerous flaw known as reentrancy. This flaw occurs when a program sends money out before it updates its own records, allowing a thief to trick the system into paying out more than it should. The researchers did not test how well the computer learns or how fast it thinks. Instead, they investigated the step that happens before the computer ever sees the data: the process of turning the source code into the map. They wanted to know if the map accurately reflected the code it was supposed to represent. Their investigation revealed a startling truth: the map-making process itself was broken in ways that no amount of training could fix.

The first problem they found was that the system was easily fooled by simple changes to the code. The researchers took a known vulnerable contract and simply renamed its parts, changing the name of the contract itself, the names of its functions, and the names of the variables that held the money. They also added lines of code that did nothing, just to clutter the file. In a normal world, these changes would make the program look different, even if the dangerous logic remained the same. However, when the researchers fed both the original and the modified versions into the system, the computer produced maps that were identical down to the last byte. The system could not tell the difference between the two. This means that an attacker could bypass the security check simply by renaming variables, without needing to know anything about how the detector works or what data it was trained on. The system was blind to the actual structure of the code, seeing only a rigid pattern of names.

Digging deeper, the researchers discovered why this happened. The tool that builds the map does not actually read and understand the code like a human programmer. Instead, it relies on a hardcoded list of forty-seven specific variable names that it is allowed to recognize. If the code uses a name from this list, the tool creates a node on the map. If the name is not on the list, the tool ignores it or creates a generic placeholder. The researchers tested this by creating four different contracts that all had the exact same dangerous flaw but used different names for the money variables. When the variable name matched the list perfectly, the map looked normal. When the name was slightly different, the map changed in confusing ways. When the name was completely new and not on the list, the map became degraded, losing important details. In the worst case, when no names matched at all, the tool fabricated parts of the map that did not exist in the original code, creating a structure based on nothing but its own internal rules. This meant that the quality of the map depended entirely on whether the programmer happened to use one of the forty-seven approved names.

The second major failure involved a specific part of the map that represents the attacker. In the system's design, there is a node meant to represent the external caller—the outside entity that triggers the attack. For a reentrancy flaw to exist, this external caller must be present in the map. The researchers found that for the most famous example of a vulnerable contract in the literature, this node was completely missing. The tool failed to draw the connection between the attacker and the vulnerable function, even though the code clearly showed it happening. This was not a limitation of the map's design, as other contracts in the training data did show this connection. It was an inconsistency in how the tool decided to build the map. Because the connection was missing, the computer had no way to see the attack pattern, even though the pattern was right there in the code.

To prove that this missing piece caused the system to fail, the researchers built a minimal, purpose-made contract that contained a fully exploitable reentrancy flaw. They used a variable name that was known to be on the approved list, ensuring that the first problem with renaming would not interfere. They fed this contract into the system. The result was a clear misclassification: the system labeled the dangerous contract as safe. The reason was simple and direct. Because the tool failed to draw the connection between the external caller and the vulnerable function, the map lacked the critical signal needed to identify the threat. The computer was not wrong in its calculation; it was simply working with an incomplete picture. The information required to make the correct decision had been discarded before the computer ever started thinking.

These findings suggest that the high accuracy numbers reported in previous studies are misleading because they were measured under conditions that did not expose these flaws. The systems performed well only because the test data happened to use the specific names and patterns the tool could recognize. In the real world, where code is written with different naming conventions and structures, these systems may fail silently. The researchers emphasize that improving the artificial intelligence model itself will not solve this problem. No amount of better training or more data can fix a map that was drawn incorrectly in the first place. The solution requires changing the way the code is converted into a map, moving away from simple name matching toward a deeper understanding of what the code actually does. Until the foundation is fixed, the security systems built on top of it remain vulnerable to the very attacks they are meant to prevent.