LegacyTranslate: LLM-based Multi-Agent Method for Legacy Code Translation

1. Problem Statement

Modernizing large-scale legacy systems in enterprise environments is a critical challenge, particularly when migrating from niche languages (like PL/SQL) to modern platforms (like Java).

• The Gap: While Large Language Models (LLMs) show promise in code translation, direct application often fails in production settings. LLMs typically generate syntactically valid code that lacks domain-specific logic, fails to align with internal architectural frameworks, and ignores shared APIs.
• The Context: The authors address a real-world migration at a European financial institution involving 2.5 million lines of PL/SQL code. The target is a modern Java architecture with strict constraints: all code must extend specific base classes and utilize internal shared libraries.
• Limitations of Prior Work: Existing benchmarks focus on popular language pairs (e.g., Python/C++) and function-level snippets. They rarely account for the framework-level constraints and API dependencies required in industrial migrations. Previous studies (e.g., Solovyeva et al.) showed LLMs could produce functional translations but were limited in scale and lacked API grounding, resulting in code that could not compile in production.

2. Methodology: The LegacyTranslate Framework

The authors propose LegacyTranslate, a multi-agent framework designed to bridge the gap between syntactic translation and production-ready code. The system decomposes the translation task into three coordinated agents, operating in a pipeline (see Figure 1 in the paper):

A. Initial Translation Agent

• Function: Generates the first draft of the Java code from the PL/SQL source.
• Mechanism: Uses Retrieval-Augmented Generation (RAG). An "Example Retriever" identifies semantically similar PL/SQL-to-Java pairs from a reference set and injects them as few-shot examples into the LLM prompt.
• Goal: To provide the LLM with context on how to translate specific logic patterns, improving structural validity.

B. API Grounding Agent

• Function: Identifies and selects the specific internal APIs required to make the code compliant with the target architecture.
• Mechanism:
  1. A curated API Knowledge Base is constructed from the target Java libraries (containing class signatures, bodies, and descriptions).
  2. If the initial code fails to compile, this agent analyzes the code and compiler errors to shortlist relevant APIs from the knowledge base.
• Goal: To ensure the generated code uses the correct internal interfaces rather than generic Java constructs.

C. Refinement Agent

• Function: Iteratively corrects the code to achieve compilation and test success.
• Mechanism: Operates in a loop. It takes the current code, the shortlisted APIs from the previous step, and compiler error messages/unit test feedback. It re-prompts the LLM to generate an improved version.
• Termination: The loop continues until the code compiles, passes all tests, or reaches a predefined limit.

3. Experimental Setup

• Dataset: An internal dataset of 68 aligned PL/SQL-to-Java pairs (manually migrated by engineers), representing diverse module types. Each sample includes unit tests executed within the actual production framework.
• Models:
  • LLM Backbone: Qwen2.5-Coder-14B (instruction-tuned for code).
  • Retriever: SFR-Mistral (dense retriever with 4096 embedding dimensions).
  • API Description Generator: Claude 3.7 Sonnet.
• Baselines:
  • LLM-only: Direct translation without examples or API info.
  • LLM + Random Examples: Translation with random (non-relevant) few-shot examples.

4. Key Results

The study demonstrates that a multi-agent approach is essential for successful enterprise migration.

Metric	LLM-only	LLM + Random Examples	LegacyTranslate (Full)
Structural Validity	42.6%	98.5%	100%
Compilation Rate	0.0%	1.5%	52.9%
Test Pass Rate	0.0%	0.0%	33.8%

Key Findings:

1. Necessity of API Grounding: Without explicit API grounding, generated code almost never compiles in the target framework (0.0% compilation for LLM-only).
2. Agent Contribution:
   • The Initial Translation Agent alone achieves a 45.6% compilation rate and 30.9% test pass rate.
   • Adding the API Grounding and Refinement Agents provides an additional 8% boost in compilation and 3% boost in test pass rate.
3. Model Capacity Matters: Larger models (Qwen2.5-14B) significantly outperform smaller models (7B, 13B). Smaller models failed to generate compilable code, likely due to insufficient capacity to handle long, context-heavy prompts and complex framework constraints.
4. Retriever Quality: High-dimensional dense retrievers (SFR-Mistral) significantly outperform smaller embedding models in retrieving relevant translation examples.

5. Key Contributions

1. LegacyTranslate Framework: A novel multi-agent system specifically designed for API-aware legacy code translation (PL/SQL to Java), addressing the gap between academic benchmarks and industrial reality.
2. Empirical Validation of Multi-Agent Architectures: Demonstrates that decomposing the task into translation, grounding, and refinement stages yields significantly higher quality than single-pass prompting.
3. Insights on Model and Retrieval Selection: Provides actionable guidance that model capacity and retrieval quality are decisive factors in handling framework-level constraints, suggesting that smaller models are insufficient for complex enterprise migrations.

6. Significance and Future Work

• Practical Impact: The paper proves that LLM-assisted migration of millions of lines of legacy code is feasible if the system is grounded in specific architectural constraints. It offers a blueprint for financial institutions and enterprises facing similar modernization challenges.
• Limitations: The current approach relies on a manually curated dataset and specific internal APIs. There is a lack of public benchmarks for realistic PL/SQL-to-Java translation.
• Future Directions: The authors suggest extending the framework to other language pairs, integrating more agents for deeper architectural analysis, and developing public datasets to facilitate broader research in this domain.