Technical Summary: Beyond Translation Accuracy: Addressing False Failures in LLM-Based Code Translation

Problem Statement

While Large Language Models (LLMs) have demonstrated significant success in automated code translation, the reliability of the evaluation frameworks used to measure this success remains under-examined. Prior work has largely focused on improving translation accuracy through advanced prompting and iterative repair, assuming that evaluation pipelines—comprising compilation commands, flags, and runtime configurations—are correct and faithful to the translated program's requirements.

This paper challenges that assumption, demonstrating that a significant number of reported failures in code translation are not due to incorrect logic or poor translation quality. Instead, these are evaluation-induced errors (false negatives) stemming from:

• Improper compilation flags (e.g., omitted libraries).
• Missing library links.
• Unconfigured or rigid runtime environments (e.g., unrealistic timeout constraints).
• Flawed test suite conversions.

These issues lead to semantically correct translations being misclassified as failures, potentially penalizing translation systems unfairly and skewing empirical conclusions about model performance.

Methodology

The authors conducted a large-scale empirical study to systematically distinguish between genuine translation errors and evaluation-induced false failures.

• Scope: The study covered 6,164 translations across five programming languages (C, C++, Java, Python, Go) and three benchmarks (Avatar, CodeNet, EvalPlus).
• Models: Translations were generated by GPT-4o, DeepSeek-Coder, and Magicoder.
• Inspection Process:
  1. Sampling: Approximately 600 initial failures were observed, from which 150 cases were manually inspected.
  2. Diagnosis: Annotators examined compilation logs, runtime traces, and test outputs. Crucially, they analyzed the evaluation configuration (compiler flags, library linking, test conversion, runtime settings) while keeping the translated code unchanged.
  3. Classification: A failure was classified as evaluation-induced if correcting the evaluation configuration alone (without modifying the code) allowed the program to compile and pass tests.
  4. Differentiation: The study distinguished between pipeline-induced failures (affecting any model under the same setup) and model-dependent failures (varying in frequency across LLMs).

Key Findings and Results

1. Pipeline-Induced Failures (Model-Independent)

These errors originate from the evaluation environment and affect all models equally if the setup is flawed:

• Missing Library Linking: Automated scripts often use generic compilation commands (e.g., gcc file.c -o file) that lack necessary linker flags. Translated C programs frequently failed due to missing flags for standard libraries like -lm (math), -lgmp (arbitrary-precision), or -lssl/-lcrypto (cryptography).
• Language-Specific Compilation Behaviors:
  • Go: Strict enforcement of unused imports and variables caused build failures when LLMs translated logic from looser languages without cleaning up redundant declarations.
  • Java/C/C++: Failures were often due to syntax violations and missing import statements.
  • Python: As an interpreted language, failures were predominantly runtime-based rather than compilation-stage errors.
• Unrealistic Timeout Constraints: Benchmarks often inherit execution time limits from highly optimized source implementations (e.g., C++). Valid translations in languages with higher runtime overhead (e.g., Python or Java) were unfairly penalized for exceeding these rigid limits.

2. Model-Dependent Failures

These issues vary in frequency across different LLMs and include:

• Natural Language Contamination: LLMs occasionally intersperse conversational fillers or Markdown formatting (e.g., ````cpp`) within the code block. If not sanitized, standard compilers interpret these as syntax errors. Magicoder showed a higher frequency (~5%) compared to GPT-4o and DeepSeek-Coder (<1%).
• Excessive Global Memory Usage: When source programs utilized large global data structures, models tended to reproduce these patterns in the target language, sometimes allocating >16GB of memory, leading to runtime failures.
• Uninvoked Entry-Point Logic: In datasets like Avatar and CodeNet, models sometimes correctly implemented logic within a function but failed to generate the main block to invoke it, resulting in no output and a test mismatch.
• Faulty PyTest-to-JUnit Conversion: In the EvalPlus benchmark, the automated conversion of test suites from Python to Java introduced errors in 5–10% of cases, often due to type mismatches or hallucinated assertions.

3. Genuine Translation Limitations

The study also identified failures that reflect inherent semantic gaps between languages, which cannot be resolved by fixing the evaluation pipeline:

• Missing Native APIs: Features like Python's eval() lack direct equivalents in Java or C++, forcing LLMs to create incomplete custom implementations.
• Divergent API Semantics: Subtle differences, such as the behavior of the modulo operator (%) with negative operands (Python retains the divisor's sign, while C/Java retain the dividend's sign), lead to silent semantic mismatches.
• Logical Corner-Case Errors: Differences in rounding logic (e.g., banker's rounding in Python vs. rounding away from zero in C/Java) and missing corner-case handling.

Contributions

The paper makes the following specific contributions:

1. Systematic Analysis: A large-scale analysis of 6,164 translations across three benchmarks and five languages, identifying a significant class of false negatives in existing benchmarks.
2. Categorization of False Failures: A clear distinction between pipeline-induced failures (configuration errors like missing flags and timeouts) and model-dependent behaviors (contamination, memory usage, entry-point logic).
3. Model Agnosticism: Confirmation that pipeline-induced issues are not artifacts of a specific model, as they were observed across GPT-4o, DeepSeek-Coder, and Magicoder.
4. Call for Transparent Standards: The identification of specific configuration flaws (missing linker flags, rigid timeouts) highlights the necessity for transparent, configuration-aware evaluation standards to accurately assess progress in LLM-based code translation.

Significance and Claims

The paper argues that the current reliance on compilation success and test case execution as primary indicators of correctness is flawed when the evaluation pipeline itself is misconfigured. By separating evaluation-induced errors from genuine translation errors, the authors demonstrate that:

• Translation systems may be unfairly penalized for issues outside their control.
• Empirical conclusions about model performance may be misleading if false negatives are not filtered out.
• Future evaluation frameworks must be "configuration-aware," detecting missing dependencies and adjusting constraints (like timeouts) based on the target language's runtime characteristics rather than blindly inheriting source-language settings.

The authors conclude that addressing these evaluation-induced errors is a prerequisite for accurately measuring the true capabilities of LLMs in code translation, rather than merely measuring the robustness of the evaluation pipeline.