💻 computer science

TRACE: Evaluating Execution Efficiency of LLM-Based Code Translation

This paper introduces **TRACE**, the first benchmark designed to evaluate the execution efficiency of LLM-translated code, revealing that functional correctness is an unreliable proxy for efficiency and that current models frequently produce inefficient translations due to algorithmic faults, language mismatches, and resource mismanagement.

Original authors: Zhihao Gong, Zeyu Sun, Dong Huang, Qingyuan Liang, Jie M. Zhang, Dan Hao

Published 2026-03-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhihao Gong, Zeyu Sun, Dong Huang, Qingyuan Liang, Jie M. Zhang, Dan Hao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a master chef who is incredibly talented at translating recipes from one language to another. If you give them a recipe for "Spaghetti Carbonara" in Italian, they can perfectly translate it into English, French, or Japanese. The ingredients list is accurate, the steps are logical, and the final dish tastes exactly like the original.

But here's the catch: What if the translation is so inefficient that it takes 10 hours to cook a dish that should take 20 minutes? Or what if the translation uses a giant industrial oven when a simple toaster would do, wasting enough electricity to power a small town?

This is the problem TRACE (Translating Code Efficiency) solves.

The Problem: "It Works, But It's Slow"

For a long time, researchers have been testing Large Language Models (LLMs) on their ability to translate code (like turning C++ code into Python). They've been asking: "Does the code run without crashing?"

If the code runs, they give it a gold star. But they've been ignoring a critical question: "How fast and how cheap is it to run?"

The paper uses a funny analogy to explain this: Imagine a translator who turns a simple instruction like "Walk to the store" into "Walk to the store, but first, take a detour to the moon, walk back, and then walk to the store." The instruction is technically correct (you still end up at the store), but it's incredibly inefficient.

The Solution: The "Stress Test" Gym

The authors built a new benchmark called TRACE. Think of TRACE as a high-intensity gym for code translators.

Small Tests vs. Stress Tests: Old benchmarks were like walking on a treadmill at a slow, steady pace. Everything looked fine. TRACE introduces "Stress Tests"—these are like sprinting up a mountain while carrying a heavy backpack.
- Example: In the paper, they translated a math algorithm. On small numbers, both the "good" and "bad" translations ran in 0.02 seconds. But when they fed the code a massive number (a stress test), the "bad" translation took 11 seconds (500 times slower!) because it made a silly mistake in the logic that only showed up under pressure.
The "Efficiency" Scorecard: TRACE doesn't just check if the code works; it measures:
- Time: How long does it take to run?
- Memory: How much RAM does it eat up?

What They Discovered

The authors tested 28 different AI models (from big tech giants like OpenAI and Google to open-source models) using this new gym. Here are the surprising findings, explained simply:

Being "Smart" Doesn't Mean Being "Fast": The AI model that was best at getting the right answer (Claude-4-think) was actually mediocre at being fast. Meanwhile, a smaller, cheaper open-source model (Qwen2.5) was often faster. It turns out, being a "genius" at logic doesn't automatically make you an "efficiency expert."
The "Translation Trap": About 1 in 4 translations that were technically correct were still terrible at performance.
- The "Wrong Tool" Problem: Sometimes the AI picks the wrong tool for the job. Imagine using a sledgehammer to crack a nut. In code terms, the AI might use a slow, complex data structure when a simple, fast one would do.
- The "Language Clash": Sometimes the AI forgets that different languages have different "idioms." It might translate a fast C++ trick into a slow, clunky Python loop, not realizing Python has a built-in shortcut for that exact thing.
Prompting Isn't a Magic Wand: The researchers tried giving the AI "cheat sheets" (prompts) telling it, "Hey, make this fast!" It helped a little bit, but not enough. It's like telling a driver to "drive faster" without teaching them how to shift gears. The AI still lacks an intrinsic sense of efficiency.

Why This Matters

In the real world, software isn't just about "does it work?" It's about "does it work well?"

Cost: Slow code costs more money in server bills.
Battery: Inefficient code drains your phone battery.
User Experience: Nobody likes an app that freezes for 10 seconds.

The Bottom Line

The paper argues that we need to stop treating code translation like a simple language exercise. We need to treat it like engineering. Just because a bridge stands up (correctness) doesn't mean it can handle heavy traffic (efficiency).

TRACE is the new ruler we need to measure not just if the code is right, but if it's good. It's a wake-up call for the AI world: "Stop just making code that works; start making code that flies."

1. Problem Statement

While Large Language Models (LLMs) have significantly improved the functional correctness of automatic code translation (migrating code between languages like C++, Java, and Python), the dimension of execution efficiency remains critically overlooked.

The Gap: Current benchmarks (e.g., TransCoder-Test) rely on small-scale unit tests that verify if code produces the correct output. They fail to detect efficiency degradations where a translation is functionally correct but algorithmically suboptimal, leading to severe runtime bottlenecks or memory bloat under stress.
The Risk: A translation that passes small tests but fails under large-scale workloads can render a migrated system impractical due to high latency, poor scalability, or excessive operational costs.
Core Challenge: Functional correctness does not guarantee efficiency equivalence. LLMs may preserve logic while introducing algorithmic faults, mismatching language constructs, or mismanaging resources.

2. Methodology: The TRACE Benchmark

The authors introduce TRACE (TRAnslated Code Efficiency), the first benchmark explicitly designed to evaluate execution efficiency in LLM-translated code. It is built upon the established TransCoder-Test dataset but enhanced through a rigorous, two-stage, LLM-driven pipeline.

A. Benchmark Construction Pipeline

Progressive Stress Test Generation:
- Instead of static tests, TRACE uses an iterative process to synthesize stress tests (inputs demanding high computational resources).
- Mechanism: An advanced LLM (GPT-4o) generates "synthesizers" (Python scripts) that produce large-scale inputs.
- Feedback Loop: The system executes these synthesizers against ground-truth reference code (C++, Java, Python). It selects inputs that maximize Execution Time (ET) and Peak Memory (PM) using a Borda Count ranking method.
- Iteration: This process repeats (up to 5 iterations), enriching the prompt with examples of previously successful high-stress generators to progressively discover more demanding edge cases.
Efficiency-Critical Task Selection:
- Not all translation tasks reveal efficiency differences (e.g., simple add functions).
- The authors filter tasks based on three criteria:
  - Feasibility: At least one LLM must produce a correct translation.
  - Impactfulness: The task must show significant resource consumption (exceeding thresholds for ET and PM) under stress.
  - Diversity: Correct translations must exhibit measurable performance variance (Coefficient of Variation > 0.05) to allow for differentiation.
- Result: The final benchmark comprises 1,000 efficiency-critical tasks across 357 problems, covering 6 translation directions (C++↔Java, C++↔Python, Java↔Python).

B. Evaluation Protocol

Models: Evaluated 28 representative LLMs (Proprietary: GPT-4o, Claude-4, O3, Gemini; Open-source: CodeLlama, DeepSeek-Coder, Qwen2.5-Coder).
Metrics:
- Correctness: Pass Rate (Pass) on default unit tests.
- Efficiency: Measured via Beyond Score ( $B_X$ ), which normalizes a model's Execution Time (ET) and Peak Memory (PM) against the best and worst performance observed in a reference set of translations for that specific task.
- Stress Testing: All correct translations are re-executed on the generated stress tests to measure efficiency degradation.

3. Key Contributions

New Dimension: First work to explicitly establish execution efficiency as a critical, orthogonal dimension to correctness in code translation.
TRACE Benchmark: A principled benchmark featuring 1,000 tasks augmented with stress tests that reveal inefficiencies invisible to small-scale tests.
Large-Scale Evaluation: Comprehensive evaluation of 28 LLMs, providing empirical evidence on the interplay between model size, reasoning capabilities, and efficiency.

4. Key Results & Findings

A. Correctness $\neq$ Efficiency

Misalignment: High correctness does not predict high efficiency.
- Example: Claude-4-think achieved the highest pass rate (95.5%) but only mid-level time efficiency. It was outperformed by smaller open-source models like Qwen2.5-Coder-14B-Instruct.
- Example: CodeLlama-34B had the lowest correctness among open-source models but produced the most time-efficient translations for the code it did get right.
Correlation: Statistical analysis shows a moderate negative correlation between correctness and time efficiency ( $r \approx -0.54$ ), suggesting models optimizing for correctness often sacrifice efficiency.

B. Inefficiency is Prevalent and Patterned

Prevalence: 23.5% of functionally correct translations are over 2 $\times$ slower or more memory-intensive than the most efficient counterparts.
Taxonomy of Errors: Manual analysis of 327 inefficient cases revealed three main categories:
1. Language Construct Mismatch (66.4%): The most common issue. LLMs map efficient source constructs to suboptimal target equivalents (e.g., using std::map [O(log N)] instead of std::unordered_map [O(1)] in C++).
2. Resource Management Inefficiency (21.7%): Unnecessary use of heavyweight abstractions (e.g., using Java's BigInteger instead of long), causing massive memory overhead.
3. Algorithm Implementation Discrepancy (11.9%): Rare but catastrophic. LLMs alter the algorithm's complexity (e.g., changing an early-exit loop to a full-range iteration), causing slowdowns of hundreds or thousands of times.

C. Directional Asymmetry

Efficiency varies significantly by translation direction. For instance, Java $\to$ Python translations are generally more efficient than Python $\to$ Java, indicating LLMs internalize language-specific idioms unevenly.

D. Limitations of Prompting

Prompt Strategies: Techniques like Few-Shot prompting (providing efficient examples) and Self-Refinement (iterative improvement) offer only modest gains.
Conclusion: Current LLMs lack intrinsic efficiency awareness. External guidance can mitigate but not eliminate the problem, suggesting a need for training paradigms that explicitly encode efficiency principles.

5. Significance

Paradigm Shift: TRACE challenges the community to move beyond "Does it work?" to "How well does it work?" in code translation.
Practical Impact: The findings warn that relying solely on correctness benchmarks can lead to the deployment of inefficient systems. Developers and researchers must now account for runtime performance when evaluating LLMs for migration tasks.
Future Research: The paper highlights that current scaling laws and reasoning techniques do not automatically solve efficiency problems, pointing toward the need for efficiency-aware fine-tuning and reinforcement learning specifically targeting runtime metrics.

In summary, TRACE provides the first rigorous framework to expose the "efficiency gap" in LLM-based code translation, demonstrating that functional correctness is an insufficient proxy for production-ready code quality.