Beyond Text Matching: Towards Reference-Free Evaluation for Human-Oriented Binary Reverse Engineering

Imagine you are trying to read a secret message written in a language that has been scrambled, stripped of its punctuation, and had all the names of the people involved erased. This is what happens when security experts try to understand computer programs that have been compiled into "binary" code—the raw, machine-readable instructions that run on your phone or laptop. Usually, these programs come with a helpful "source code" manual, but when hackers or malware authors hide their tracks, that manual disappears. Experts have to use special tools to translate the scrambled binary back into something that looks like code, but it often comes out looking like a messy, confusing jumble of numbers and generic names like "sub_401B20." To make sense of this, they need to rewrite it into a human-friendly story, a process called "Human-Oriented Binary Reverse Engineering." But here's the tricky part: how do you know if the new, cleaned-up story is actually good? You can't just ask a computer to count how many words match a perfect answer, because the perfect answer doesn't exist (the original manual is gone). And asking a human expert to read every single line is too slow, too expensive, and they get tired. So, the big question is: Can we teach a super-smart AI to act as a judge and tell us if the translation is good, without needing the original manual to compare it against?

This paper dives into that exact question, exploring whether Large Language Models (LLMs)—the same kind of AI that can write poems or solve math problems—can be the "referees" for these messy code translations. The researchers found that these AI judges are indeed much better at understanding the meaning of the code than old-school computer metrics, which just look for matching words. In fact, the AI judges agreed with human experts about 63.20% of the time, whereas the old computer methods only agreed about 35.04% of the time. However, the paper also discovered a surprising twist: there is no single "best" AI setup that works for every situation. Sometimes a specific AI model with a specific way of asking questions works great for naming functions, but fails miserably at summarizing code. It's like trying to use one pair of shoes for running, swimming, and hiking; you need the right gear for the right job.

To solve this, the authors built a clever system called "BinJudge." Think of it as a smart traffic controller that looks at each specific piece of code and instantly picks the perfect AI judge and the best way to ask it the question. This system doesn't just pick one "best" judge for everyone; it adapts on the fly. By doing this, BinJudge managed to get even closer to human experts (improving agreement by up to 24.7%) while also saving a massive amount of money—costing only a tiny fraction (between 0.06 and 0.84 times) of what it would cost to use the most expensive, static setup. The paper proves that while we can't just pick one magic button to fix everything, we can build a smart, flexible system that makes evaluating these tricky code translations fast, cheap, and surprisingly accurate.

The Background: Decoding the Scrambled Message

Before we get to the main event, let's set the stage with a few key ideas.

Binary Reverse Engineering is like trying to rebuild a cake after it's been baked, flattened, and had all the frosting scraped off. You have the final product (the binary file), but you don't have the recipe (the source code). Experts use tools to reverse the process, turning the machine code back into something that looks like programming code. However, because the original "recipe" is gone, the result is often messy.

Human-Oriented Binary Reverse Engineering (HOBRE) is the next step. It's the art of taking that messy, machine-like code and polishing it up so a human can actually read it. This involves giving functions cool, descriptive names (instead of "sub_401B20"), writing clear summaries of what the code does, and fixing the structure so it looks like a well-written story.

The Evaluation Problem: How do you know if the "polishing" worked?

• The Old Way (Text Matching): Imagine grading a student's essay by counting how many words match a teacher's answer key. If the student says "The cat ran fast" and the key says "The feline sprinted quickly," the old computer says, "Wrong! Zero points!" because the words don't match. This is bad for code because there are many ways to say the same thing.
• The Human Way: A human expert reads the code and says, "Yep, that makes sense." This is the "gold standard," but it's slow, expensive, and hard to scale.
• The New Idea (LLM-as-a-Judge): What if we ask an AI, which has read millions of books and code snippets, to act as the teacher? Instead of just counting words, the AI understands the meaning. It knows that "cat" and "feline" are the same, and it can tell if a summary actually explains the code or just sounds fancy.

The Paper's Journey: Testing the AI Judges

The researchers, led by Xiuwei Shang and their team, decided to put this "AI Judge" idea to the test. They didn't just guess; they built a massive, high-quality playground called BinJudgeBench.

Building the Playground:
They took 51 real-world software projects, compiled them into binary code, and then stripped away all the helpful names and comments. They then asked 8 different AI models to try and fix the code back up. To know who did a good job, they had three human experts (who are basically code detectives) read the results and grade them on a scale of 1 to 5. They checked for things like:

• Does it make sense? (Semantic Correctness)
• Does it help a human understand the code faster? (Utility)
• Is it written in a natural, human style? (Idiomization)

This created a "Gold Standard" dataset with over 1,200 carefully graded examples.

The First Discovery: AI Judges vs. Old Metrics
The team compared the AI judges against the old "word-counting" metrics. The results were clear: the AI judges were much better at understanding the human side of things.

• The AI judges agreed with the human experts 63.20% of the time.
• The old computer metrics only agreed 35.04% of the time.

This suggests that AI can actually "get" the meaning of the code, not just the spelling. It's like the difference between a robot that counts how many times you say "hello" and a friend who understands that you are actually greeting someone warmly.

The Second Discovery: The "One-Size-Fits-All" Myth
Here is where it gets interesting. The researchers tried different settings for the AI judges:

• Different Models: Some AIs are huge and expensive (like GPT-4o), others are smaller and cheaper.
• Different Prompts: Some were asked to just give a score (Zero-Shot), others were shown examples of what a "5" or a "1" looks like (Few-Shot), and some were asked to explain their reasoning step-by-step (Chain-of-Thought).
• Different Temperatures: This controls how "creative" or "random" the AI is when answering.

They found that there is no single best setup.

• For some tasks, like summarizing code, showing the AI examples (Few-Shot) worked best.
• For other tasks, like naming functions, asking the AI to think step-by-step (Chain-of-Thought) helped smaller models do better.
• Sometimes, a huge, expensive model was the best; other times, a smaller, cheaper model was just as good.

This means you can't just pick one "best" AI and use it for everything. It's like trying to use a sledgehammer to fix a watch; sometimes you need a tiny screwdriver, and sometimes you need a heavy hammer.

The Solution: BinJudge (The Smart Traffic Controller)
Since there's no single best setup, the team built BinJudge. This is a lightweight system that acts like a smart router.

1. It looks at a specific piece of code.
2. It asks itself: "Which AI model and which question style will work best for this specific piece of code?"
3. It picks that specific combination and sends the code to that judge.

The Results of BinJudge:

• Better Accuracy: By picking the right judge for the job, BinJudge improved the agreement with human experts by 4.5% to 24.7% compared to just using a single, static setup.
• Cheaper: Because it often picks smaller, cheaper models when they are good enough, it reduced the cost to between 0.06x and 0.84x of the cost of using the most expensive "best" setup.

What This Means

The paper concludes that we don't need to rely on expensive human experts for every single piece of code, nor do we have to settle for dumb computer metrics that just count words. We can use AI judges, but we have to be smart about it. We can't just pick one "best" AI; we need a system that adapts, choosing the right tool for the right job.

The authors are careful to note that while AI judges are great, they aren't perfect. They sometimes get confused if the code is too vague or if the AI gets too "fluent" but wrong. But overall, this work suggests that with the right adaptive system, we can evaluate code translations in a way that is fast, cheap, and surprisingly close to what a human expert would say. It's a big step toward making the dark, messy world of binary code a little more readable for everyone.