Technical Summary: Beyond Code Pairs: Dialogue-Based Data Generation for LLM Code Translation

Problem Statement

Large Language Models (LLMs) have demonstrated significant capabilities in code translation for popular languages like Python and Java. However, their performance deteriorates significantly in low-resource programming domains (e.g., Fortran) and specialized frameworks (e.g., CUDA), where high-quality parallel training data is scarce. Existing agent-based approaches typically follow a source-to-target workflow, producing final source–target code pairs. While effective for generating syntactically valid code, these methods discard valuable intermediate information—such as compiler errors, runtime diagnostics, and interactions with external tools—that is crucial for enabling deeper reasoning about program semantics and functional behavior. Consequently, models trained on static code pairs often fail to recover from errors or generate functionally correct code in complex translation scenarios.

Methodology

The authors propose an automated dataset generation pipeline that moves beyond traditional code pairs by incorporating a dual-LLM Questioner–Solver design within an agent-based translation workflow. This approach generates multi-turn dialogues that capture the reasoning process behind translation refinement.

The Questioner–Solver Module

The core of the system consists of two distinct LLM roles:

• Questioner: Analyzes the current state, leverages dialogue memory (past attempts, errors, fixes), and incorporates external signals (compiler errors, runtime outputs). It formulates targeted questions, specifies constraints, and checks for source–target consistency.
• Solver: Executes translations, generates unit tests, proposes fixes, and returns verification results.

Multi-Turn Dialogue Generation Pipeline

The pipeline processes Source Programming Language (SPL) code through the following stages to generate verified dialogues:

1. Preprocessing: Filters source code to ensure it is self-contained, executable, and fits within context windows.
2. SPL Unit Test Generation & Verification: The Solver generates unit tests for the source code. The Questioner analyzes execution results; if failures occur, the Solver repairs the tests or code. This loop continues until tests pass (up to a threshold).
3. Initial Translation: The Questioner requests a target-language translation, and the Solver produces the initial code.
4. Translation Verification & Refinement: The translated code is compiled and executed against the unit tests. The Questioner evaluates results and instructs the Solver to refine the translation based on errors. This iterative loop captures problem-solving trajectories.
5. Duo-Test Verification: A final stage ensures functional equivalence between source and translated programs by comparing outputs. Successful translations and their full dialogue histories are stored as verified data.

Dataset Construction

The pipeline produces two types of data formats for fine-tuning:

• Dialogue-Level: Full multi-turn conversations capturing the entire workflow.
• Question–Solution (QS) Pairs: Atomic exchanges extracted from dialogues, decomposing the reasoning into fine-grained steps.

The authors instantiated this pipeline for two tasks: Fortran→C++ and C++→CUDA, generating 3.64k and 3.93k dialogues, respectively. These were further decomposed into over 14.1k (Fortran→C++) and 12.7k (C++→CUDA) QS pairs.

Key Contributions

1. Automated Dialogue Generation Pipeline: A framework driven by a dual-LLM Questioner–Solver module that integrates external knowledge (compilers, runtimes) into structured reasoning traces, moving beyond static code pairs.
2. New Translation Benchmarks: Large-scale datasets of verified translations with unit tests for Fortran→C++ and C++→CUDA, accompanied by three disjoint test suites for rigorous evaluation.
3. Validation of Data Structuring Strategies: The authors demonstrate that different data formats optimize for different tasks. Decomposing dialogues into fine-grained QS-Pairs is highly effective for mastering syntactic complexity, while using the full Dialogue trace is superior for preserving semantic correctness.

Experimental Results

The authors evaluated the approach by fine-tuning mid-size open-weight models (CodeLlama-13B, DeepSeek-Coder-6.7B, Qwen2.5-7B) on the generated data.

• Functional Correctness: Fine-tuning on the generated data yielded dramatic improvements. On the challenging C++→CUDA task, the CodeLlama-13B model's unit test success rate increased from 12.5% to 68.8% (a >56% boost) when trained on QS-Pair data. Similarly, Fortran→C++ translation saw CodeLlama-13B improve from 42.3% to 74.1%.
• Debug-Round Gains: Models trained on dialogue data showed consistent improvements across multiple debug rounds, surpassing models trained on static code pairs in compilation, execution, and unit-test success.
• Competitiveness with Proprietary Systems: A dialogue-tuned Qwen2.5-7B model significantly outperformed larger proprietary systems (Gemini 2.5 Flash, Llama 4 Scout 17B) on key metrics like compilation and execution success, highlighting the cost-efficiency of dialogue-centered supervision.
• CodeBLEU Scores: The fine-tuned models demonstrated consistent improvements in CodeBLEU scores, indicating better syntactic and structural alignment with reference solutions.

Significance

The paper claims that dialogue-driven supervision consistently improves functional correctness across multiple open-weight model families, particularly in low-resource domains where traditional parallel corpora are unavailable. By embedding structured reasoning (compiler feedback, error correction steps) into the dataset, the approach enables LLMs to learn not just the end result of translation, but the process by which correctness is achieved. The results suggest that this method provides a cost-effective pathway for achieving state-of-the-art performance in legacy code modernization, allowing smaller open-weight models to rival or exceed larger proprietary systems. The authors note that while the framework focuses on functional correctness as a necessary first step, future work will aim to incorporate performance-aware evaluation metrics.