JMigBench: A Benchmark for Evaluating LLMs on Source Code Migration (Java 8 to Java 11)
The paper introduces JMigBench, a benchmark designed to evaluate the ability of large language models to migrate Java 8 code to Java 11, revealing that while models like Mistral Codestral can handle simple API substitutions, they still struggle with complex migrations and cannot yet replace human developers.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
The "Old House Renovation" Problem: A Simple Guide to JMigBench
Imagine you own a beautiful, classic house built in 1990 (this is Java 8). It’s sturdy and works great, but the world has changed. The plumbing is outdated, the electrical wiring doesn't meet modern safety codes, and the old heating system is becoming impossible to repair. To keep the house safe and functional, you need to upgrade it to 2024 standards (this is Java 11).
In the world of software, this is called "Code Migration." It is a massive, boring, and exhausting chore for human programmers. They have to go through thousands of lines of code and swap out old, "deprecated" parts for new, modern ones.
The New "Robot Contractor"
Lately, people have been wondering: "Can we just hire an AI robot to do this renovation for us?"
The researchers in this paper wanted to test if a specific AI "robot" (called Mistral Codestral) is actually good at being a renovation contractor, or if it’s just a robot that looks like it knows what it’s doing.
The Problem: The "Messy Toolbox"
To test the robot, you need a very specific set of tasks. You can't just give it a whole house; you need to give it specific, broken parts to fix.
The researchers first tried to find real-world "broken parts" from the internet, but it was like trying to find specific broken lightbulbs in a giant junkyard—everything was mixed up, and many of the parts weren't actually broken; they were just old.
So, they did something smarter: they hand-crafted a "Test Kit" (which they named JMigBench). They created 45 perfect examples of "old parts" (Java 8) and paired them with the "correct new parts" (Java 11). This gave them a clean, fair way to grade the robot.
The Test Results: The "Good with Paint, Bad with Pipes" Rule
The researchers gave the AI robot the Test Kit and watched it work. Here is what they found:
- The "Easy Fixes" (The Paint Job): If the task was simple—like swapping an old-fashioned doorknob for a modern one—the robot was great! It got some of these perfectly right. It understood the "look" of the new code.
- The "Complex Fixes" (The Plumbing & Wiring): When the task got complicated—like replacing a complex central heating system (specifically things called CORBA or JAX-WS)—the robot completely froze. It didn't know how to handle the deep, structural changes. It would look at the old, broken part and just... leave it there.
The Scorecard:
- Perfect Score: The robot only got the job 100% right in about 11% of the cases.
- The "Leftover" Problem: In many cases, the robot "fixed" the code but forgot to remove the old, dangerous parts, meaning the "house" (the software) might still be unsafe.
The Bottom Line
The paper concludes that while these AI robots are getting smarter, they aren't ready to take over the construction site just yet.
The takeaway: You can use the AI to help you with the "easy stuff" (like painting the walls or changing lightbulbs), which saves you some time. But when it comes to the heavy-duty structural work, you still need a human expert to make sure the house doesn't fall down!
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.