💬 NLP

Beyond BLEU: A Semantic Evaluation Method for Code Translation

This paper proposes a novel semantic evaluation methodology for code translation that utilizes compiler testing to measure execution correctness, demonstrating that LLM-based decompilers significantly outperform heuristic approaches while traditional syntactic metrics like BLEU fail to correlate with functional accuracy.

Original authors: Julius Näumann, Sven Keidel, Amir Molzam Sharifloo, Mira Mezini

Published 2026-05-08

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Julius Näumann, Sven Keidel, Amir Molzam Sharifloo, Mira Mezini

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to translate a recipe from a foreign language into English.

The Old Way: Counting Words
Traditionally, when computers check if a translation is good, they use a method called BLEU. Think of this like a teacher grading a translation by simply counting how many words match the original, ignoring the meaning.

The Problem: If the original recipe says "Add a pinch of salt," and the translation says "Add a tiny bit of salt," the computer thinks they are very different because the words don't match perfectly.
The Bigger Problem: If the original says "Bake at 350°F" and the translation accidentally says "Bake at 500°F," the computer might still give them a high score because the words look similar. The result? You get a burnt cake, but the computer says, "Great job!"

The authors of this paper argue that for computer code, this "word-counting" method is useless. Two programs can look completely different on the page but do the exact same thing (like two different ways to tie a shoe). Conversely, two programs can look almost identical but do completely different things (like a recipe that says "add sugar" vs. "add salt").

The New Way: The Taste Test
Instead of just looking at the words, the authors propose a "Taste Test" approach. They want to see if the translated code actually works the same way as the original.

Here is how their new method works, using a creative analogy:

The Generator (The Chef): They use a tool called Csmith to automatically cook up thousands of random, simple computer programs. It's like a robot chef making thousands of random, tiny dishes.
The Reference Dish: They run these original programs and measure a specific "flavor profile" (a mathematical checksum) of the result. This is their baseline.
The Translation: They feed the original code to a translator (either a human-like AI or a traditional rule-based tool) to get the "translated" code.
The Re-Cooking: They take that translated code, cook it again, and measure the new "flavor profile."
The Verdict: If the flavor profiles match perfectly, the translation is Semantically Correct. It means the code does the exact same job, even if the words look different. If the flavors don't match, the translation failed, even if the words looked similar.

What They Found
They tested this method on two types of translators:

The Old School (Heuristic): These are traditional tools that follow strict rules.
The New AI (LLM): A large language model trained to translate code.

The Results:

The AI Wins: The AI translator was much better at keeping the "flavor" (the actual function) of the code intact compared to the old rule-based tools.
The Old Metric is Broken: When they looked at the "word-count" scores (BLEU), they found zero connection to whether the code actually worked.
- Sometimes the AI got a low word-match score but the code worked perfectly.
- Sometimes the AI got a high word-match score, but the code was broken.

The Bottom Line
The paper concludes that we need to stop grading code translations by how much they look like the original. Instead, we must grade them by whether they act like the original. The old way of counting words is like judging a car by how much it looks like a Ferrari, while the new way is actually driving it to see if it runs. The authors show that the "driving test" reveals that AI is getting much better at this task, while the "look-alike" test is misleading us.

Technical Summary: Beyond BLEU: A Semantic Evaluation Method for Code Translation

Problem Statement
Code translation, particularly decompilation (translating low-level binaries to higher-level representations), is a core capability of Large Language Models (LLMs). However, evaluating the correctness of these translations remains a significant challenge. Current evaluation standards rely heavily on syntactic similarity metrics like BLEU, CodeBLEU, and RUBY. These metrics measure surface-level string resemblance or Abstract Syntax Tree (AST) structure but fail to capture program semantics.

The paper identifies two critical flaws in using string similarity for semantic-preserving tasks:

False Negatives: Functionally identical programs may receive low similarity scores due to syntactic variations, identifier choices, or compiler optimizations (e.g., reordering instructions or constant folding).
False Positives: Programs with high textual similarity can exhibit divergent behaviors. For instance, a decompiler might syntactically resemble the source but incorrectly interpret an instruction (e.g., misinterpreting a lea instruction as a memory load), leading to semantic errors despite a high BLEU score.
Furthermore, existing test-based approaches require manual test case creation, which is labor-intensive and rarely exhaustive.

Methodology
To address these limitations, the authors propose a scalable evaluation methodology inspired by compiler testing techniques, specifically utilizing Csmith (Yang et al., 2011b). The approach shifts the focus from string comparison to semantic equivalence validation via execution.

The process involves the following steps:

Program Generation: Csmith is used to automatically generate random, compilable C programs performing arbitrary computations and function calls. This eliminates manual effort and allows for the creation of arbitrarily large evaluation sets.
Reference Execution: The generated C programs are compiled to binaries at optimization levels -O0 and -O3. During execution, these binaries compute and record a CRC checksum over intermediate values.
Translation and Re-execution: The resulting binaries are processed by the decompiler under evaluation (e.g., an LLM or heuristic tool) to produce source code. This decompiled code is then recompiled and executed.
Semantic Validation: The checksum of the re-executed program is compared against the reference checksum. A mismatch indicates a semantic error in the translation.
Metric Definition: The authors introduce a Semantic Correctness Score, defined as the proportion of translated programs that produce matching checksums upon execution:
$\text{Semantic Score} = \frac{\# \text{programs w/ correct checksum}}{\# \text{tested programs}}$

Key Contributions
The paper presents three primary contributions:

A Novel Evaluation Methodology: A framework based on automated program generation and semantic checksum validation, applicable to any language but implemented here for C.
A Direct Metric: The Semantic Correctness Score, which provides a direct measure of functional accuracy rather than syntactic resemblance.
Empirical Evidence: A demonstration that syntactic metrics (BLEU) are poor predictors of functional accuracy in code translation tasks.

Experimental Results
The methodology was applied to evaluate Meta's LLMCompiler (a 13B parameter model fine-tuned for binary lifting from x86 assembly to LLVM bytecode) against state-of-the-art heuristic decompilers: RetDec and McToll.

Performance Comparison: LLMCompiler significantly outperformed heuristic approaches. It achieved semantic correctness scores of 0.33 (on -O0 binaries) and 0.63 (on -O3 binaries). In contrast, neither McToll nor RetDec produced any binaries that successfully executed; McToll failed to lift any binaries, and RetDec produced binaries that either failed to compile or crashed at runtime.
Correlation Analysis: The study computed round-trip BLEU scores for LLMCompiler outputs and compared them against execution results. The results showed negligible correlation between BLEU scores and semantic correctness.
- For -O0 binaries, the point-biserial correlation ( $r$ ) for BLEU-1 was -0.13.
- For -O3 binaries, the strongest predictor, CodeBLEU, showed a correlation of only 0.35.
- Mean similarity scores for passing and failing test cases were nearly identical, indicating that syntactic similarity distributions overlap significantly regardless of functional correctness.

Significance and Claims
The paper claims that syntactic metrics like BLEU are fundamentally unsuited for assessing semantic-preserving code translation. The findings demonstrate that high string similarity does not guarantee functional accuracy, and conversely, functional accuracy does not require high string similarity.

The authors position their work as a necessary shift toward execution-based evaluation. While they acknowledge limitations—specifically that the framework is currently restricted to C-generated programs and that the generated code may lie outside the training distribution of the evaluated models—they assert that the framework provides a valuable, rigorous measure of performance. A high semantic score indicates strong applicability and semantic equivalence, suggesting that robust models should generalize beyond their specific training data.

Technical Summary: Beyond BLEU: A Semantic Evaluation Method for Code Translation

More like this