💻 computer science

TRACE: Evaluating Execution Efficiency of LLM-Based Code Translation

This paper introduces **TRACE**, the first benchmark designed to evaluate the execution efficiency of LLM-translated code, revealing that functional correctness is a poor proxy for efficiency and that current models frequently produce inefficient code due to algorithmic faults, language mismatches, and resource mismanagement.

Original authors: Zhihao Gong, Zeyu Sun, Dong Huang, Qingyuan Liang, Jie M. Zhang, Dan Hao

Published 2026-03-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhihao Gong, Zeyu Sun, Dong Huang, Qingyuan Liang, Jie M. Zhang, Dan Hao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you hire a master translator to translate a complex recipe from French to English. The translator does a perfect job: the ingredients are correct, the steps make sense, and if you follow the English version, you get a delicious cake. Functionally, the translation is perfect.

But here's the catch: The French recipe said, "Stir the pot gently for 1 minute." The English translation says, "Stir the pot gently for 1 minute," but the translator accidentally added a hidden instruction: "Before you start, walk around the kitchen 10,000 times to warm up your muscles."

You still get the cake, but it takes you 10 hours instead of 10 minutes. You wasted energy, time, and money.

This is exactly the problem TRACE (the subject of this paper) is trying to solve in the world of Artificial Intelligence.

The Problem: "Correct but Slow"

For a while, we've been asking AI models (Large Language Models or LLMs) to translate code from one programming language (like C++) to another (like Python). We've been very happy if the code works. If the AI produces code that runs without crashing and gives the right answer, we give it a gold star.

But the researchers behind this paper realized: Just because it works doesn't mean it's good.

They found that AI models often translate code in a way that is functionally correct but incredibly inefficient. It's like the AI translated the recipe but forgot to remove the "walk around the kitchen" step. In the real world, this means software that runs 500 times slower or eats up 30 times more computer memory than it needs to.

The Solution: The TRACE Benchmark

To fix this, the authors created TRACE. Think of TRACE as a stress test gym for code translators.

The Old Way: Previous tests were like a "light jog." They gave the AI simple problems (like adding two small numbers) to see if the code worked. The AI passed easily.
The TRACE Way: TRACE is a marathon with a heavy backpack. It takes those same problems but creates "stress tests"—massive, difficult inputs that push the computer to its limits.
- Analogy: If the old test asked, "Can you lift a 5lb weight?" TRACE asks, "Can you lift a 500lb weight while running a sprint?"
- Suddenly, the AI that looked perfect on the light jog is struggling to breathe under the heavy load.

What They Discovered

The researchers put 28 different AI models through this gym. Here are the three big things they found:

1. Being "Smart" Doesn't Mean Being "Efficient"
The AI models that got the highest scores for "correctness" (the ones that got the most gold stars) were often the slowest runners.

Analogy: The most famous, expensive translator (Claude-4-think) was great at grammar but terrible at speed. Meanwhile, a smaller, less famous open-source model (Qwen2.5) was actually much faster and more efficient.
Lesson: A correct translation is not a guarantee of a good translation.

2. The Mistakes Follow Patterns
The researchers looked at why the code was slow. They found three main "culprits":

The Algorithm Swap (12%): The AI changed the math entirely. Instead of using a fast shortcut, it used a slow, roundabout way to solve the problem.
The Wrong Tool (66%): This was the most common mistake. The AI picked the wrong "tool" for the job.
- Analogy: Imagine the original code used a screwdriver (fast and precise). The AI translated it to use a hammer (it gets the job done, but it's clumsy and slow). In programming, this happens when an AI uses a slow data structure (like a tree) when a fast one (like a hash map) would have been perfect.
The Heavy Backpack (22%): The AI added unnecessary weight. It used giant, heavy objects to do simple tasks, wasting memory.
- Analogy: Using a massive, industrial-sized truck to deliver a single letter.

3. "Prompting" Only Helps a Little
The researchers tried to fix the problem by giving the AI better instructions (like saying, "Please be efficient!").

Analogy: It's like telling a runner, "Try to run faster!" It helps a tiny bit, but it doesn't change the fact that the runner is wearing heavy boots. The AI doesn't intrinsically understand efficiency; it just guesses.

Why This Matters

This paper is a wake-up call. For years, we've been celebrating AI code translation just because it "works." But in the real world, efficiency is everything.

If you build a website or an app using AI-translated code that is 500% slower, your users will wait forever, your servers will crash, and your electricity bill will skyrocket.

TRACE is the first tool that forces us to stop asking, "Does it work?" and start asking, "Does it work well?" It ensures that the code we generate isn't just a working recipe, but a recipe that actually gets dinner on the table in time.

1. Problem Statement

While Large Language Models (LLMs) have significantly advanced the functional correctness of automatic code translation (migrating code between languages like C++, Java, and Python), the dimension of execution efficiency remains critically overlooked.

The Gap: Existing benchmarks (e.g., TRANSCODER-TEST) rely on small-scale unit tests that verify functional correctness but fail to expose performance bottlenecks.
The Risk: An LLM can produce code that passes all functional tests but is orders of magnitude slower or more memory-intensive than the source code due to algorithmic misunderstandings, suboptimal language idioms, or poor resource management.
Motivation: As software complexity grows, inefficient translations can lead to severe runtime bottlenecks, scalability issues, and increased operational costs, rendering functionally correct translations impractical.

2. Methodology: The TRACE Benchmark

The authors introduce TRACE (TRAnslated Code Efficiency), the first benchmark explicitly designed to evaluate execution efficiency in LLM-translated code.

A. Construction Pipeline

TRACE is built upon the established TRANSCODER-TEST benchmark (568 problems) using a two-stage, LLM-driven pipeline:

Progressive Stress Test Generation:
- Instead of static tests, the authors use an iterative loop to synthesize stress tests (inputs demanding high computational resources).
- Mechanism: An advanced LLM (GPT-4o) generates "test input synthesizers" (Python scripts that programmatically create inputs).
- Feedback Loop: The system executes these synthesizers against ground-truth reference code. Based on execution profiles (time and memory), the LLM is prompted with examples of the most demanding inputs to generate even more challenging tests in subsequent iterations.
- Selection: Tests are ranked using the Borda Count method across C++, Java, and Python to select those that consistently maximize resource consumption.
Efficiency-Critical Task Selection:
- Not all translation tasks exhibit efficiency variance. The authors filter the dataset to retain only tasks where:
  - Feasibility: At least one model produces a correct translation.
  - Impactfulness: The task has non-trivial resource costs (exceeding specific thresholds for time/memory).
  - Diversity: Correct translations exhibit significant performance variance (Coefficient of Variation > 0.05), ensuring the task can differentiate between efficient and inefficient models.

B. Benchmark Statistics

Scope: 357 problems resulting in 1,000 efficiency-critical tasks across six translation directions (C++↔Java, C++↔Python, Java↔Python).
Test Suite: Each task includes 10 default correctness tests and 10 stress tests.
Impact: Stress tests increase average execution time by 8.9× and peak memory usage by 3.4× compared to default tests, effectively exposing latent inefficiencies.

3. Key Contributions

New Dimension: Establishes execution efficiency (time and memory) as a critical, orthogonal dimension to functional correctness in code translation.
TRACE Benchmark: Provides the first principled foundation for efficiency-oriented evaluation, featuring stress-tested inputs and efficiency reference sets.
Large-Scale Evaluation: Evaluates 28 representative LLMs (including proprietary models like GPT-4o, Claude-4, and open-source models like Qwen2.5-Coder and CodeLlama) to provide empirical insights into the state of the art.

4. Key Results & Findings

A. Correctness $\neq$ Efficiency

Misalignment: Functional correctness is a poor proxy for efficiency. The model with the highest pass rate (Claude-4-think, 95.5%) achieved only mid-level time efficiency, outperformed by smaller open-source models like Qwen2.5-Coder-14B-Instruct.
Correlation: Statistical analysis shows a moderate negative correlation between correctness and time efficiency ( $r \approx -0.54$ ), suggesting that models optimizing for correctness may inadvertently sacrifice efficiency.

B. Inefficiency is Prevalent and Patterned

Prevalence: 23.5% of functionally correct translations are significantly inefficient (over 2× slower or more memory-intensive than the best alternative).
Taxonomy of Errors: A manual analysis of 327 inefficient cases revealed three primary categories:
1. Language Construct Mismatch (66.4%): Using suboptimal data structures or APIs (e.g., using std::map $O(\log N)$ instead of std::unordered_map $O(1)$ in C++).
2. Resource Management Inefficiency (21.7%): Unnecessary overhead, such as using BigInteger in Java for primitive integer operations.
3. Algorithm Implementation Discrepancy (11.9%): Fundamental algorithmic errors, such as changing loop bounds or complexity (e.g., $O(N)$ to $O(N^2)$ ).

C. Limitations of Current Strategies

Prompting: Inference-time strategies (Zero-shot, Few-shot, Self-refinement) yield only modest improvements. While Few-shot prompting helps slightly, it cannot eliminate the lack of intrinsic efficiency awareness in current LLMs.
Scaling & Reasoning: Larger models and reasoning-enhanced variants (e.g., "Think" models) do not reliably improve efficiency. In some cases, smaller models or standard chat models outperform their larger/reasoning counterparts in efficiency metrics.
Directional Asymmetry: Efficiency varies significantly by translation direction. For example, models are generally more efficient translating Java $\to$ Python than Python $\to$ Java, indicating uneven internalization of language-specific idioms.

5. Significance and Future Implications

Paradigm Shift: The paper argues that code translation evaluation must move beyond "does it run?" to "how well does it run?"
Benchmark Standard: TRACE sets a new standard for evaluating LLMs, forcing developers to consider computational complexity and resource usage during translation.
Research Direction: The findings suggest that current training paradigms are correctness-driven. Future work must explicitly encode efficiency principles and language-specific idioms into LLM training to bridge the gap between functional correctness and practical performance.
Real-World Impact: By identifying that 1 in 4 correct translations are inefficient, TRACE highlights a critical risk in adopting LLMs for legacy modernization and code migration, where performance degradation could be catastrophic at scale.