Technical Summary: Revisiting the Role of Natural Language Code Comments in Code Translation

1. Problem Statement

Automated code translation between programming languages (PLs) remains a significant challenge for Artificial Intelligence. While Large Language Models (LLMs) offer a scalable alternative to traditional rule-based transpilers, they frequently struggle to generate accurate and reliable translations, even for self-contained programs. Existing research has focused on reducing translation bugs by providing context, yet the specific role of natural language code comments in improving translation quality remains inadequately characterized.

A critical gap exists in current benchmarks: code comments are largely absent from evaluation datasets. Since most code-specific LLMs are pre-trained on well-commented repositories (e.g., GitHub), it is hypothesized that augmenting source code with natural language descriptions could aid translation. However, without systematic empirical study, the impact of comments—whether beneficial or detrimental—remains unknown.

2. Methodology

The authors conducted a large-scale empirical study involving over 80,000 code translations to evaluate the impact of comments.

Experimental Setup

• Datasets: The study utilized 1,100+ unique code samples from two established benchmarks: AVATAR (Java, Python) and CodeNet (C, C++, Go). These samples were augmented with model-generated comments.
• Languages: Pairwise translations were performed across five programming languages: C, C++, Go, Java, and Python.
• Models:
  • Translating LLMs: CodeLlama-13B, DeepSeek-Coder-V2, GPT-4o-mini, Granite-8B-Code-Instruct, and StarCoder-1.
  • Commenting LLMs: DeepSeek-Coder-V2, GPT-4o-mini, and Mistral 7B were used to generate comments for uncommented source code.
• Evaluation Metric: A translation was deemed successful if the generated code compiled, passed runtime checks, and satisfied all existing test cases.

Research Questions (RQs)

The study addressed four specific research questions:

1. Usefulness: Do natural language comments improve translation performance?
2. Intent: Can comments be classified by intent (e.g., descriptive, explanatory), and does intent affect utility?
3. Density & Language: How do comment density and the language of the comment (English vs. others) influence accuracy?
4. Location: Does the placement of comments (e.g., pseudo-code at the start vs. line-specific comments) impact performance?

3. Key Findings and Results

RQ1: Usefulness of Code Comments

The study revealed a non-monotonic relationship between comments and translation quality.

• Variable Impact: Comments can significantly enhance performance (up to +435% gain in specific cases) but can also severely degrade it (down to -90% loss).
• Noise vs. Clarity: While comments often resolve compilation and test-case failures by clarifying logic, indiscriminate injection of comments can introduce noise or constrain the LLM's translation logic, leading to new errors.
• Model Dependency: The effectiveness of comments depends heavily on the specific pairing of the commenting model and the translating model. For instance, DeepSeek and GPT generally produced more beneficial comments than Mistral.

RQ2: Intent of Code Comments

The authors created CJBench, a dataset of 200 author-written Java programs, to analyze comment intent.

• Descriptive vs. Complex: Short, descriptive comments (explaining what the code does) were found to be the most beneficial.
• Noise from Complexity: Author-written comments often contained multiple intents (explanatory, precautionary, informative, analytical). The study found that these multi-intent, longer comments often performed worse than concise, model-generated descriptive comments. The presence of non-essential intents appeared to create noise that translating LLMs struggled to filter.

RQ3: Density and Language

• Density: Arbitrarily restricting comment density (e.g., commenting only the top 1/3 of complex lines) did not yield better results than commenting the entire code. The study suggests that without error-localization heuristics, selective commenting is difficult to optimize.
• Language: English comments consistently outperformed comments in other languages (Chinese, French, Japanese) for translation tasks, with only marginal exceptions.

RQ4: Location and Format

The study compared three forms of natural language specifications:

1. Pseudo-code (appended to the prompt).
2. Method Specifications (placed at method boundaries).
3. Code Comments (placed alongside specific lines of code).
   Result: Line-specific code comments were the most effective approach, outperforming both pseudo-code and method specifications in most scenarios.

4. Proposed Solution: COMMENTRA

Based on the finding that comments are beneficial but risky if used indiscriminately, the authors propose COMMENTRA, an iterative translation framework.

• Mechanism:
  1. Iteration 1: Attempt translation on the original, uncommented code.
  2. Iteration 2: For samples that fail, inject comments (using a commenting LLM like DeepSeek) into the source code and re-attempt translation.
  3. Iteration 3: For samples still failing, inject comments using a different, potentially higher-quality commenting model (e.g., GPT) and re-attempt.
• Efficiency: This approach is cost-effective, as expensive models are only invoked for samples that failed previous, cheaper iterations.
• Performance: COMMENTRA demonstrated the ability to potentially double the performance of LLM-based code translation, with average gains ranging from 23% to 71% across two iterations.

5. Significance and Claims

The paper claims to be the first comprehensive study of its scale and language coverage regarding the impact of code comments on LLM-based code translation.

• Empirical Evidence: It provides strong evidence that comments describing the overall purpose of code (rather than line-by-line functionality) significantly enhance translation accuracy.
• Practical Framework: The proposed COMMENTRA approach offers a simple, low-cost, and easily integrable method to improve translation accuracy without retraining models.
• Nuanced Insight: The study challenges the assumption that "more context is always better," demonstrating that the quality, intent, and placement of comments are critical factors.

The authors conclude that while comments are a powerful tool for improving code translation, their utility is highly dependent on how they are generated and applied. The iterative, targeted approach of COMMENTRA effectively mitigates the risks of noise while capitalizing on the clarifying power of natural language.