RepoGenesis: Benchmarking End-to-End Microservice Generation from Readme to Repository
RepoGenesis introduces the first multilingual benchmark for end-to-end microservice repository generation from READMEs to full codebases, revealing that while current agents and IDEs achieve high API coverage and deployment success, they still struggle with architectural coherence and cross-file consistency, a gap that can be addressed by fine-tuning models on this dataset.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are a chef who has just been handed a detailed recipe card for a complex, multi-course banquet. Your goal isn't just to chop a single onion or fry an egg (which is what most current AI coding tests ask); your goal is to build the entire restaurant from scratch, including the kitchen layout, the plumbing, the menu, the waitstaff, and the actual food, all while ensuring the lights turn on and the customers can actually eat.
This is exactly what the paper RepoGenesis is about.
Here is the breakdown of the research in simple terms:
1. The Problem: The "Lego Brick" vs. The "Castle"
For a long time, AI coding assistants have been tested on small tasks.
- Old Tests: "Write a function that adds two numbers" or "Fix this one broken line of code." This is like asking a robot to build a single Lego brick.
- The Reality: Real software development is like building a whole castle. You need to design the architecture, manage the dependencies (the mortar between bricks), and ensure every room connects perfectly.
- The Gap: Existing AI tools are great at making bricks, but they often fail when asked to build the whole castle. They might build a beautiful tower that collapses because the foundation is missing, or they build a kitchen that has no plumbing.
2. The Solution: RepoGenesis (The "Grand Challenge")
The researchers created a new benchmark called RepoGenesis. Think of this as a massive, rigorous cooking competition for AI.
- The Task: The AI is given a "Recipe Card" (a README file describing a web service) and must generate a complete, working software repository. This includes all the code, configuration files, and instructions on how to start the server.
- The Scale: They created 106 different "recipes" (repositories) ranging from simple tasks (like a "Rock, Paper, Scissors" game) to complex systems (like a file management service or a user authentication platform).
- The Languages: They tested both Python (like a flexible, quick-cooking kitchen) and Java (like a strict, industrial-grade factory).
- The Quality Control: Before the competition started, the researchers used a "Review-Rebuttal" process. Imagine three expert judges tasting the test dishes, arguing about the score, and having a head chef (a human) make the final call to ensure the tests were fair and perfect.
3. The Competition: Who Showed Up?
They pitted the best AI coding agents and commercial IDEs (like GitHub Copilot, Cursor, and Antigravity) against these challenges.
The Results were shocking:
- The "Deployability" Paradox: The AI was surprisingly good at starting the restaurant. Many systems could get the lights on and the server running (high "Deployment Success Rate").
- The "Functionality" Failure: However, once the restaurant opened, the food was often inedible. The AI frequently failed to actually implement the features correctly.
- The Score: Even the best AI (Copilot with a top-tier model) only got about 23% of the tasks perfectly right on the first try.
- The Analogy: It's like an AI that can build a car that starts and drives down the street, but the brakes don't work, the radio is missing, and the seats are made of cardboard.
4. Where Did They Fail? (The "Why")
The researchers analyzed the failures and found three main reasons the AI struggled to build the castle:
- Cross-File Consistency (The "Lost Sock" Problem): The AI would write code in one file that said "Use the Blue Door," but in another file, it would write "Use the Red Door." The files didn't talk to each other correctly.
- Architectural Coherence (The "Bad Blueprint"): The AI would design a system where the kitchen was on the roof and the dining room was in the basement. The structure made no sense.
- Dependency Management (The "Missing Ingredients"): The AI would write a recipe calling for "Saffron," but forget to tell the system to buy the Saffron. The code looked right, but it crashed when it tried to run.
5. The Silver Lining: Fine-Tuning Works
The researchers took an open-source model (Qwen3-8B) and "trained" it specifically on their new benchmark data.
- The Result: This specialized AI, called GenesisAgent, performed almost as well as the most expensive, top-tier commercial models (like GPT-5 mini).
- The Lesson: This proves that if you give AI the right kind of "homework" (high-quality, real-world examples), it can learn to build entire software systems, not just write snippets.
Summary
RepoGenesis is a wake-up call for the AI industry. It shows that while AI is getting very good at writing small pieces of code, it is still struggling to be a Software Architect that can design, build, and deploy a complete, working system from scratch.
The paper provides the "exam" (the benchmark) and the "study guide" (the dataset) to help AI get better at building the whole castle, not just the bricks.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.