JETO-Bench: A Reproducible Benchmark for Execution Time Improvement Patches in Java
This paper introduces JETO-Mine, a novel configurable tool for generating reproducible benchmarks of execution time improvement patches in Java, which was used to create JETO-Bench—a dataset of 660 identified patches from 174 repositories—and demonstrated that current coding agents like OpenHands achieve a 14.3% success rate on these tasks while highlighting a significant lack of performance-related tests in open-source projects.
Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you have a massive library of software code written in Java, a popular programming language. For years, researchers have been trying to build "robot mechanics" (AI agents) that can automatically find bugs in this code and fix them. However, most of these robots have only been trained to fix functional bugs—mistakes where the software crashes or gives the wrong answer (like a calculator saying 2+2=5).
What about performance bugs? These are mistakes where the software works correctly but is incredibly slow, like a car engine that runs fine but sputters and takes forever to reach 60 mph. Until now, it has been very hard to test if our robot mechanics can fix these "slow" problems, especially in Java.
Here is a simple breakdown of what this paper introduces:
1. The Problem: The "Volatility" of Java
Fixing speed issues in Java is like trying to time a runner on a track that changes its surface every few seconds.
- The Challenge: Java has a special "engine" (the JVM) that gets faster the more you use it (Just-In-Time compilation) and occasionally stops to clean up its workspace (Garbage Collection). This makes measuring speed very tricky. A test might look slow just because the computer was "warming up," not because the code is bad.
- The Gap: Existing benchmarks (test sets) for fixing speed issues mostly exist for other languages like Python or C++. Java was missing a fair, reliable test set.
2. The Solution: JETO-Mine (The "Gold Miner")
The authors built a tool called JETO-Mine. Think of this as a high-tech gold panning machine for software code.
- Phase 1: The Search (Static Analysis): The machine scans millions of commits (changes) on GitHub. It uses a smart AI (an LLM) to read the "notes" developers left when they made changes, looking for clues that say, "I made this run faster."
- Phase 2: The Lab (Dynamic Analysis): This is the most important part. Once a potential "speed fix" is found, JETO-Mine puts the code into a Docker container. Think of this as a sealed, identical laboratory for every single test.
- It runs the code 30 times to get an average.
- It uses strict math (statistical testing) to prove the speed-up is real and not just a fluke caused by the computer's random noise.
- It ensures the "before" and "after" versions run in the exact same environment so the comparison is fair.
- Phase 3: The Judge (Evaluation Harness): This is a referee that checks if a robot mechanic's fix actually works. It runs the new code, checks if it still passes all the original tests, and measures if it is truly faster.
3. The Result: JETO-Bench (The "Treasure Chest")
Using JETO-Mine, the researchers created JETO-Bench.
- They dug through 11 years of history and nearly 1.8 million code changes.
- They found 660 potential speed fixes.
- After rigorous testing, they verified 91 of these as "gold standard" fixes that are guaranteed to run and be reproducible.
- Key Discovery: They found that most open-source Java projects do not have tests that prove a piece of code is faster. It's like having a race car but no stopwatch to prove it's fast. This is a huge gap in how software is currently tested.
4. The Test Drive: Can AI Fix These Bugs?
To see if JETO-Bench is useful, the researchers took a leading AI coding agent called OpenHands and asked it to fix these 91 speed problems.
- The Score: OpenHands successfully fixed 14.3% of the issues.
- The Meaning: This result is similar to what other studies found for Python and C++. It shows that while AI is getting better, fixing "slow" code is still very hard.
- The Bonus: The "Judge" (evaluation harness) automatically caught almost all the wrong fixes. If the AI tried to fix the wrong part of the code or broke the build, the system knew immediately.
5. Why This Matters
- Reproducibility: Before this, if you wanted to test a new speed-fixing tool, you had to hope the original code still worked years later. JETO-Bench provides a sealed, time-capsule environment (Docker) so anyone can run the exact same test today and get the same result.
- A New Challenge: The paper highlights that we need to invent new ways to write tests that measure speed, not just correctness.
- No Magic Bullet Yet: The study confirms that even advanced AI agents struggle with performance optimization, suggesting this is a frontier that needs more research.
In a nutshell: The authors built a machine (JETO-Mine) to find and verify real-world examples of Java code getting faster. They packaged these examples into a test set (JETO-Bench) and proved that current AI robots can fix about 1 in 7 of these speed problems, but they also revealed that the software world is missing the tools (tests) to properly measure speed improvements in the first place.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.