Specification-Driven Code Translation Powered by Large Language Models: How Far Are We?
This paper investigates using natural language specifications as an intermediate representation for code translation with Large Language Models, finding that while combining them with source code yields improvements for specific language pairs like Python and C++, the approach does not provide consistent overall performance gains.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are trying to translate a complex recipe from one language (like French) to another (like Japanese). You have the original recipe (the Source Code), but you know that sometimes, just looking at the French words isn't enough to get the Japanese translation perfect. You might need a middle step: a simple, plain-English description of what the recipe is supposed to do (the NL-Specification).
This paper asks a big question: Does adding that "plain English" middle step actually help Large Language Models (LLMs) translate code better?
Here is the breakdown of their findings, using everyday analogies:
The Setup: Three Ways to Translate
The researchers tested three different ways to ask an AI to translate code:
- The Direct Approach: "Here is the French recipe. Translate it to Japanese." (Source Code Only)
- The Summary Approach: "Here is a plain English summary of the recipe. Now write the Japanese version." (NL-Specification Only)
- The Hybrid Approach: "Here is the French recipe and the plain English summary. Translate it to Japanese." (Source + NL-Specification)
They tested this with many different "languages" (Python, C++, Java, etc.) and different AI models.
The Big Surprise: The Middleman Isn't Always Helpful
You might think that having a clear, simple summary would always help. It didn't.
- The "Summary Only" Trap: When the AI tried to translate only from the plain English summary (without seeing the original code), it often failed. It was like trying to rebuild a complex machine just from a verbal description; you lose the specific details (like the exact size of a screw or the order of operations). The AI lost the "structural context" needed to get it right.
- The "Hybrid" Sweet Spot: Adding the summary alongside the original code helped in some specific cases, but not all.
- When it worked: It was like a helpful tour guide. For complex, messy code (especially in Python and C++), the summary acted as a "denoising filter." It helped the AI ignore the confusing syntax and focus on the main idea.
- When it failed: For languages that are very strict or verbose (like C or Java), the summary often threw away critical low-level details (like memory management). In these cases, the summary actually made the translation worse because it hid important rules.
The Verdict: The "Direct Approach" (Source Code Only) was actually the most reliable overall. The summary is a "complementary signal"—it helps fix specific problems, but it's not a magic bullet that works everywhere.
The "Garbage In, Garbage Out" Problem
The researchers found that the quality of the translation depends entirely on the quality of the summary.
- The Good Scenario: If the AI generates a perfect, accurate summary, the translation improves.
- The Bad Scenario: If the AI makes a tiny mistake in the summary (like getting a math step wrong), that mistake gets baked into the final code. The AI faithfully follows the wrong instructions, just like a chef following a flawed recipe.
The "Repair Shop"
A major part of the study was fixing errors. They found that many translation failures were just "typos" or small syntax errors (like missing a semicolon).
- They built a system that acts like a spell-checker for code. If the translation fails to compile, the AI is asked to fix the specific error.
- Result: This simple "repair step" fixed a huge number of errors (improving accuracy by about 6–8%). It showed that the AI often understands the logic but just messes up the grammar.
The Quality Check (SonarQube)
Finally, they checked the "hygiene" of the translated code using a tool called SonarQube (which scans for security holes and bad practices).
- The Finding: The translation method didn't drastically change how "clean" the code was.
- The C-Language Spike: However, they noticed that translating into the C language resulted in significantly more security warnings (like unsafe memory access). It seems the AI struggles to follow the strict safety rules of C, regardless of whether it used a summary or not.
Summary for the General Audience
Think of this research as testing a new translation strategy for a team of robots.
- Old Way: Robots look at the original text and guess the translation.
- New Way: Robots read a summary first, then translate.
- Conclusion: The summary helps when the original text is confusing, but it often hurts when the text is very technical or precise. The best strategy is usually to stick to the original text, but use the summary as a backup plan for tricky spots. And, just like a human editor, the robots need a "spell-check" step at the end to catch their small mistakes.
The paper concludes that while using natural language summaries is an interesting idea, it doesn't replace the need for the original code. It's a helpful tool, but only when used carefully and in the right situations.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.