RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades
RoadmapBench introduces a benchmark of 115 complex, long-horizon coding tasks based on real open-source version upgrades to demonstrate that current frontier AI agents struggle significantly with large-scale, multi-target software development, resolving fewer than 40% of tasks even with the most advanced models.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are hiring a team of super-smart, AI-powered junior developers. You want to know if they can handle a real job: not just fixing a single typo in a recipe, but completely rewriting a massive, 500-page cookbook to add new chapters, change how ingredients are measured, and fix broken ovens—all while keeping the old recipes working for people who still use the old version.
That is exactly what the paper ROADMAPBENCH is about.
Here is the breakdown of the paper in simple terms:
1. The Problem: The "Typo" vs. The "Remodel"
For a long time, we tested AI coding assistants on small, easy tasks. It's like asking, "Can you fix this one sentence in this paragraph?" The AI usually says, "Yes!" and gets an A+.
But in the real world, software development isn't about fixing one sentence. It's about renovating a skyscraper. You have to change the plumbing, the electrical wiring, and the elevator systems across 50 different floors, all at once, without the building collapsing.
The authors realized that existing tests were too easy. They were like asking an architect to fix a leaky faucet, when the real job is designing a new wing for a hospital. So, they built a new, much harder test called ROADMAPBENCH.
2. The New Test: The "Version Upgrade" Challenge
Instead of asking the AI to fix a bug, ROADMAPBENCH gives it a roadmap.
- The Setup: The AI is dropped into a digital workshop with an old version of a real software project (like a popular open-source tool).
- The Mission: The AI is handed a "roadmap" document. This document says, "In the next version of this software, we need to add these 5 new features, change how these 3 things work, and fix these 2 bugs."
- The Scale: This isn't a tiny change. On average, the AI has to rewrite 3,700 lines of code across 51 different files. It's like asking the AI to rewrite the entire script of a movie, change the costumes, and re-shoot the scenes, all in one go.
- The Rules: The AI cannot peek at the "answer key" (the new version of the software). It has to figure it out from the instructions alone.
3. The Results: The AI is Still a "Junior"
The researchers tested 13 of the smartest AI models available (including the very latest versions from companies like Anthropic, OpenAI, and Google). They put these models through the ROADMAPBENCH test.
The outcome was sobering:
- Even the smartest AI (Claude-Opus-4.7) only succeeded in 39.1% of the tasks.
- The weakest AI succeeded in only 5.2% of the tasks.
The Metaphor:
If you asked a human junior developer to remodel a house, and they only finished the job correctly 4 times out of 10, you would say, "They are still learning." The paper shows that even our most advanced AI is still in the "learning" phase when it comes to complex, long-term software projects.
4. Where Do They Fail?
The paper didn't just count the failures; it looked at why they failed. They found a pattern based on how "smart" the AI was:
- The Stronger Models: They usually got the code to compile (build) and could write most of the features. But they failed on the details. It's like they built a perfect house, but the door handle was on the wrong side, or the light switch didn't actually turn on the light. They understood the big picture but missed the tiny, precise logic.
- The Weaker Models: They often couldn't even get the house built. They forgot to install the roof, or they tried to put the kitchen in the basement. They failed at the basic construction level.
5. Why This Matters (According to the Paper)
The authors argue that we need to stop testing AI on "typo fixes" because it gives us a false sense of security. Just because an AI can fix a bug doesn't mean it can build a new feature.
ROADMAPBENCH is a new, honest ruler. It tells us that while AI is getting better, the ability to handle long, complex, multi-step software projects is still a huge, unsolved challenge. The AI is currently like a very talented apprentice who can follow instructions well for a few steps, but gets confused when the project gets too big and complex.
Summary
- Old Tests: "Can you fix this one broken word?" (AI: Yes!)
- ROADMAPBENCH: "Here is a plan to upgrade a whole software system. Can you do it?" (AI: I can try, but I'll probably mess up the details or forget a step.)
- Conclusion: We are not there yet. The AI is smart, but it isn't ready to be the lead engineer on a major software project just yet.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.