DevOps-Gym: Benchmarking AI Agents in Software DevOps Cycle
This paper introduces DevOps-Gym, the first end-to-end benchmark comprising 700+ real-world tasks across 30+ Java and Go projects to evaluate AI agents in full DevOps workflows, revealing that current state-of-the-art models struggle significantly with complex tasks like issue resolving, test generation, monitoring, and build configuration.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you have a brilliant new apprentice who is incredibly good at writing individual sentences and paragraphs. You ask them to write a story, and they do a fantastic job. But now, you want to see if they can run an entire publishing house. This involves not just writing the story, but also setting up the printing press, watching the machines run to see if they overheat, fixing the machine when it jams, and writing a checklist to make sure the next book doesn't have the same jam.
This is exactly what the paper DEVOPS-GYM is testing.
The Big Idea: From "Writer" to "Factory Manager"
For a while, AI has been great at writing code (like the apprentice writing sentences). But real software isn't just about writing; it's about the whole lifecycle, known as DevOps. This includes:
- Building: Getting the code to compile and run.
- Monitoring: Watching the running program to see if it's acting weird (like a machine overheating).
- Fixing: Debugging and patching problems.
- Testing: Making sure the fix actually works and doesn't break anything else.
The researchers built DEVOPS-GYM, a giant "gym" or training ground with over 700 real-world challenges. They wanted to see if AI agents could act like a full-time Factory Manager who handles all four of these steps, not just the writing part.
The "Gym" Setup
The researchers created a realistic environment using real software projects written in Java and Go (two languages common in big companies, unlike Python which is often used for AI training).
They designed four types of workouts for the AI:
- The "Setup" Workout (Build & Config): The AI has to take a messy project and get it to build correctly. This is like trying to assemble a complex piece of furniture without the instructions, or switching from one brand of tools to another.
- The "Surveillance" Workout (Monitoring): The AI has to watch a running program and spot subtle problems, like a memory leak (where the program slowly eats up all the computer's memory) or a slow network connection. It's like a security guard who has to notice a slow leak in a pipe before the basement floods.
- The "Repair" Workout (Issue Resolving): The AI has to find a bug described in a report and fix the code.
- The "Quality Check" Workout (Test Generation): The AI has to write a test to prove the bug is gone.
They even created End-to-End Pipeline tasks, which are like a relay race. The AI has to finish the Setup, then the Surveillance, then the Repair, and finally the Quality Check, all in one go without dropping the baton.
The Results: The Apprentice is Still a Novice Manager
The researchers tested the smartest AI agents available today (using models like Claude, OpenAI's o4-mini, and others). Here is what they found:
- The "Setup" is Hard: Even the best AI struggled to get projects to build correctly. They often got confused by the complex rules of how to compile code. The best agent only succeeded about 52% of the time.
- The "Surveillance" is a Nightmare: This was the biggest failure. The AI was terrible at watching a running system. They often failed to notice problems or got distracted. The success rate was shockingly low, often 0% to 20%. It's like a security guard who falls asleep or looks at the wrong camera.
- The "Repair" is Language Dependent: The AI was good at fixing bugs in Python (which it has seen a lot of in training), but when they switched to Java and Go, the performance dropped significantly. It's like a translator who speaks perfect French but stumbles when asked to translate a technical manual in German.
- The Relay Race Failed Completely: When asked to do all four steps in a row, 0% of the AI agents succeeded. They couldn't keep the whole picture in their head. They might fix the build, but then forget to check the monitoring, or fix the bug but fail to write the test.
Why Did They Fail?
The paper suggests a few reasons why these "Factory Managers" aren't ready yet:
- They don't know the tools: The AI hasn't been trained enough on how to use specific system tools (like checking memory usage or managing build files).
- They get distracted: When watching a system over time, the AI loses track of what it's looking at. It can't handle the "long story" of a system running for hours.
- They lack deep knowledge: They can fix a simple typo, but they don't understand the deep, complex rules of how big software systems are built and run.
The Bottom Line
DEVOPS-GYM is a reality check. While AI is amazing at writing code snippets, it is currently not ready to run the entire software factory on its own. It struggles with the messy, complex, and long-term tasks of building, watching, fixing, and testing real-world software. The researchers say we need to do a lot more work before AI can truly automate the full software development cycle.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.