💻 computer science

MigrationBench: Repository-Level Code Migration Benchmark from Java 8

This paper introduces MigrationBench, a comprehensive repository-level benchmark and evaluation framework for migrating Java 8 code to modern LTS versions (17 and 21), demonstrating that an agentic LLM framework can achieve high success rates on this challenging task.

Original authors: Linbo Liu, Xinle Liu, Qiang Zhou, Lin Chen, Yihan Liu, Hoan Nguyen, Behrooz Omidvar-Tehrani, Xi Shen, Jun Huan, Omer Tripp, Anoop Deoras

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Linbo Liu, Xinle Liu, Qiang Zhou, Lin Chen, Yihan Liu, Hoan Nguyen, Behrooz Omidvar-Tehrani, Xi Shen, Jun Huan, Omer Tripp, Anoop Deoras

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, old library filled with books written in a specific, slightly outdated style of English (let's call it "English 8"). The library is huge, containing thousands of books, and every single one of them is currently in perfect condition.

Now, imagine the library board decides to upgrade the entire building to a new, modern standard ("English 17"). This isn't just about changing the font; it's about updating the grammar rules, the vocabulary, and the way the books are organized so they work with the new building's security systems and elevators.

MigrationBench is a giant challenge created by researchers at AWS AI to test how well Artificial Intelligence (specifically, Large Language Models or LLMs) can handle this massive renovation project.

Here is the breakdown of their work using simple analogies:

1. The Problem: Moving a Whole City, Not Just a House

Most previous AI tests were like asking a robot to fix a single leaky faucet or write a single paragraph. But real-world software isn't just one file; it's a whole city of interconnected buildings (a "repository").

The Challenge: Moving from "English 8" to "English 17" requires changing thousands of files at once. If you change one street sign, you might break the traffic lights three blocks away.
The Gap: Until now, there was no standardized "exam" to see if AI could handle renovating an entire city of code without crashing the whole system.

2. The Solution: The "MigrationBench" Dataset

The researchers built a massive test suite called MigrationBench.

The Full Dataset: They gathered 5,102 real, open-source Java projects from the internet (like GitHub). They filtered them to ensure they were high-quality, had proper licenses, and were currently working perfectly.
The "Selected" Subset: Testing all 5,000+ projects takes forever. So, they hand-picked 300 of the most complex, tricky, and interesting projects. Think of this as the "Final Exam" version of the dataset, designed to really stress-test the AI.

3. The Rules of the Game (Evaluation)

How do you know if the renovation was successful? The researchers set up two levels of difficulty:

Level 1: Minimal Migration (The "It Works" Test)
- Goal: The AI just needs to make sure the code compiles (builds) and all the existing tests pass in the new version.
- Analogy: The building is safe, the lights turn on, and the elevators work. The books are readable.
- Success Rate: With their best AI setup, they achieved a 71.67% success rate here.
Level 2: Maximal Migration (The "Modernization" Test)
- Goal: The AI must not only make it work but also upgrade every single tool and library inside the code to their absolute latest, most secure versions.
- Analogy: Not only is the building safe, but the AI also replaced all the old plumbing with new copper pipes, updated the security cameras to 4K, and installed the latest smart-home features. This is much harder because new tools often have different rules.
- Success Rate: This is much tougher. The best AI setup achieved a 53.33% success rate.

4. The Workers: AI Agents

The researchers didn't just ask the AI to "write code." They built AI Agents—digital workers equipped with tools.

The Basic Worker: A simple AI that looks at the code and tries to fix errors. It struggled with the "Maximal" upgrade.
The Smart Worker (Prompt Engineering): They gave the basic worker a better set of instructions, telling it specifically to upgrade everything. This helped a lot.
The Researcher Worker (RAG): They gave the worker a "phone book" (a knowledge base) of the latest software versions so it didn't have to guess. This improved the results further.
The Hybrid Team (The Winner): This was the cleverest approach. They used a computer program to automatically swap out the old tools for new ones (a fast, robotic step) and then let the AI Agent come in to fix the messy parts where the new tools didn't fit perfectly.
- Result: This team did just as well as the most expensive AI worker but used 11% less computing power.

5. The Results

The paper shows that while AI is getting very good at fixing small code problems, renovating an entire software "city" from an old version to a new one is still a massive challenge.

The Hybrid approach (combining automated tools with AI) proved to be the most efficient way to do it.
The dataset and the code for these AI agents are now public, so other researchers can try to build even better renovation teams.

In short: MigrationBench is a new, rigorous gym for AI to practice moving entire software libraries from old versions to new ones. It shows that while AI is capable, the most complex "renovations" still require a smart mix of automated tools and human-like reasoning to get the job done right.

Technical Summary: MigrationBench

Problem Statement

While Large Language Models (LLMs) have demonstrated significant success in code generation and isolated issue resolution, their capabilities in repository-level code migration remain under-evaluated. Unlike tasks focusing on single functions or files, code migration requires a holistic approach to navigate interconnected issues across an entire codebase, particularly when upgrading legacy systems. Specifically, there is a lack of large-scale, high-quality benchmark datasets designed to assess the migration of Java 8 repositories to modern Long-Term Support (LTS) versions (Java 17 and 21). Existing benchmarks often focus on code generation, translation between languages, or small-scale maintenance tasks, failing to capture the complexity and scale of real-world migration scenarios involving thousands of lines of code and complex dependency trees.

Methodology

1. Dataset Curation (MigrationBench)

The authors introduce MigrationBench, the first large-scale benchmark for Java repository migration. The curation process involved a rigorous, multi-step filtering pipeline applied to GitHub repositories:

Initial Filtering: Repositories were filtered by license (MIT/Apache 2.0), quality (minimum 3 stars), and build tool (Maven).
Compatibility Verification: The team identified a "base commit" ( $H_b$ ) for each repository where the code successfully builds and passes tests under Java 8. This involved verifying pom.xml configurations, ensuring no hard-coded Java 8 versions were incompatible with the target, and validating that compiled class major versions were 52 (Java 8).
De-duplication: Repositories were de-duplicated based on exact matches of file hashes and content.
Test Requirement: Repositories were partitioned into a Full dataset (5,102 repositories with unit tests) and a UTG subset (4,814 repositories without tests). The Full dataset serves as the primary benchmark for migration evaluation.
Selected Subset: A representative subset of 300 repositories (selected) was curated from the Full dataset. This subset emphasizes larger, multi-module repositories to better reflect the diversity and intricacy of real-world migration challenges.

2. Evaluation Framework

Due to the absence of ground-truth target repositories, the authors propose an automated evaluation framework based on Approximate Functional Equivalence (FE):

Minimal Migration: Defined by four criteria:
1. Build Success: mvn clean verify passes under Java 17.
2. Version Validation: Compiled classes have a major version of 61 (Java 17).
3. Test Method Invariance: Test methods (annotated with @Test) are not renamed, disabled, or removed (verified via AST parsing).
4. Test Count Invariance: The number of test cases does not decrease.
Maximal Migration: Builds upon minimal migration by adding a fifth criterion:
1. Dependency Modernization: All dependencies are upgraded to their latest stable major versions available on Maven Central (as of November 2024).

3. Agentic Frameworks

The paper evaluates several agent-based approaches using the Strands agent framework and Claude-4.5-Sonnet:

Baseline (Strands Agent): Equipped with shell and edit tools to migrate code and ensure mvn clean verify passes.
Prompt Engineering (PE) Agent: Augmented with system prompts explicitly instructing the agent to upgrade all dependencies to their latest major versions.
RAG Agent: Incorporates a Retrieval-Augmented Generation approach where the agent queries an external dependency knowledge base to identify correct latest versions.
Hybrid Approach: Combines static code analysis (deterministically upgrading dependencies) with the LLM agent. The static tool handles version upgrades, and the agent fixes resulting build failures and compatibility issues.

Key Contributions

MigrationBench Dataset: The introduction of the first large-scale benchmark for repository-level code migration, comprising 5,102 repositories (Full) and a curated subset of 300 repositories (Selected).
Evaluation Framework: A comprehensive, automated framework for assessing migration success through minimal and maximal criteria, addressing the lack of ground truth in migration tasks.
Agentic Solutions: The proposal and evaluation of an agentic workflow for code migration, including a novel Hybrid Approach that combines static analysis with LLM reasoning to improve efficiency.
Open Resources: The release of the dataset, evaluation source code, and detailed agent trajectories (including intermediate decisions and failure modes) to support reproducibility and future research.

Experimental Results

Experiments were conducted on the selected subset (300 repositories) using the Strands agent framework with Claude-4.5-Sonnet.

Minimal Migration: The baseline Strands agent achieved a 71.67% success rate (pass@1) for minimal migration.
Maximal Migration:
- The baseline agent achieved only 15.33% success.
- Adding Prompt Engineering (PE) increased efficacy to 45.67%.
- Incorporating RAG further improved efficacy to 53.33%.
- The Hybrid Approach matched the RAG agent's 53.33% efficacy while reducing the average number of LLM calls per repository (52.55 vs. 59.22), demonstrating a favorable cost-efficiency trade-off.
Model Comparison: When applying the Hybrid Approach to other models (DeepSeek-V3.1, Qwen3-Coder-480B, GLM-5), Claude-4.5-Sonnet remained the most effective, though other models showed varying degrees of success (e.g., GLM-5 achieved 45.33%).
Compilation vs. Verification: Disabling unit tests (relying only on mvn clean compile) increased minimal migration efficacy from 71.67% to 97.67%, highlighting the critical role of test suites in validating functional equivalence.

Significance and Claims

The paper positions MigrationBench as a critical resource for advancing research in software engineering and LLM capabilities. The authors claim that:

Complexity Gap: Migration tasks present a significantly higher level of difficulty than code generation or isolated issue resolution, requiring holistic repository understanding.
Feasibility: LLMs, particularly when guided by agentic workflows and external knowledge (RAG) or hybrid static analysis, can effectively tackle repository-level migration from Java 8 to Java 17.
Standardization: The proposed evaluation framework provides a rigorous, standardized method for assessing migration success, moving beyond simple compilation checks to include dependency modernization and test preservation.
Future Direction: The release of agent trajectories and the UTG subset opens new avenues for studying agent limitations, failure modes, and the generation of unit tests for legacy code.

The authors acknowledge limitations, including the reliance on existing test coverage (which may not guarantee full functional equivalence for untested code) and potential reproducibility issues due to Maven Central throttling during large-scale data collection. However, they maintain that MigrationBench offers a robust foundation for systematic and reproducible evaluation of code migration methods.