SCDBench: A Benchmark for LLM-Based Smart Contract Decompilers
This paper introduces SCDBench, a comprehensive benchmark dataset and evaluation methodology designed to rigorously assess LLM-based smart contract decompilers by revealing that while frontier models can generate compilable code, they currently struggle to achieve semantic consistency, with the best model correctly decompiling only 42 out of 600 real-world contracts.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you have a delicious, complex cake (a Smart Contract) that someone baked and locked inside a sealed, opaque box. You can see the box, and you can even shake it to hear the ingredients rattle around (the Bytecode), but you can't see the recipe.
In the world of blockchain, this "recipe" is the original code written by the developer. Unfortunately, most of these boxes are never opened, and the recipes are lost. To understand what's inside or if the cake is safe to eat, we need to decompile it: to reverse-engineer the box and write down the recipe again.
For a long time, we tried to do this with old, rigid tools that produced messy, hard-to-read notes. Recently, we started using super-smart AI assistants (Large Language Models, or LLMs) to do the job. These AIs are great at reading the "rattle" of the box and writing a recipe that looks perfect and even works in a kitchen. But here's the problem: How do we know if the AI actually recreated the exact same cake, or just a cake that looks like one?
This is where the paper SCDBench comes in.
The Problem: The "Fake Cake" Trap
The authors explain that existing tests for these AI decompilers are like judging a cake by its smell alone. They might look at the recipe the AI wrote and say, "Hey, that looks like a valid cake recipe!" But they don't actually bake it and taste it to see if it matches the original.
With the new, super-smart AIs, this is dangerous. An AI could write a recipe that:
- Looks perfect on paper.
- Can be baked (compiles successfully).
- Has the right number of eggs and flour (matches the interface).
- But tastes completely different (the logic is wrong).
If you trust that fake recipe, you might lose your money or security. We need a way to test if the AI's recipe produces the exact same result as the original.
The Solution: SCDBench (The Ultimate Taste Test)
The authors built a new "Taste Test" called SCDBench. It's a standardized challenge with 600 real-world "boxes" (smart contracts) that they already know the recipes for.
They created a 4-Step Ladder to grade the AI's performance. The AI has to climb every rung to get a passing grade:
- Format Check (Did you follow instructions?): Did the AI write the recipe in the right format? Did it include the oven temperature and the name of the cake? If the recipe is messy or missing parts, it fails here.
- Compilability (Can you bake it?): Does the recipe actually work in a real kitchen? If you try to bake it, does it explode, or does it come out as a solid cake? Many AIs write recipes that look good but are impossible to bake.
- Interface Recovery (Do you have the right ingredients?): Does the recipe call for the exact same ingredients as the original? If the original cake had a "chocolate" layer and the AI's recipe forgot it, or added a "spicy" layer that wasn't there, it fails.
- Semantic Consistency (The Taste Test): This is the big one. The authors take the AI's recipe, bake the cake, and then taste it against the original. They run specific "taste tests" (inputs) on both cakes. If the AI's cake reacts differently (e.g., it melts when the original stays firm), the AI fails. This is the hardest step.
The Results: AIs are Good, But Not Perfect
The authors tested three top-tier AI models (Claude Opus, GPT-5, and GLM-5) on this challenge. Here is what they found:
- The AIs are getting better at writing recipes: The top models can write recipes that look perfect and can even be baked (compiled) most of the time.
- The "Repair" Trick: When the AI wrote a recipe that almost worked but had a tiny typo, the researchers let the AI try to fix it once. This "self-repair" step helped a lot, turning many broken recipes into working ones.
- The Taste Test is Still Hard: Even the best AI models failed the final taste test for most contracts.
- Out of 600 contracts, the best model only perfectly recreated the logic for 42 of them (less than 7%).
- For the hardest contracts, the success rate was even lower.
The Big Takeaway
The paper concludes that while AI is amazing at making smart contracts look and compile correctly, it still struggles to understand the deep, hidden logic that makes them work exactly the same way as the original.
Think of it like this: The AI is a brilliant chef who can copy the appearance of a dish perfectly, but it's still learning how to replicate the exact flavor of a secret family recipe. Until the AI can pass the "Taste Test" (Semantic Consistency) consistently, we can't fully trust it to reverse-engineer these digital contracts for security or transparency.
SCDBench is the new standard tool that forces these AIs to prove they aren't just faking it, by making them pass this rigorous, four-step taste test.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.