CI-Repair-Bench: A Repository-Aware Benchmark for Automated Patch Validation via CI Workflows
This paper introduces CI-Repair-Bench, a repository-aware benchmark constructed from real GitHub Actions executions that evaluates automated program repair exclusively through full CI re-execution, revealing that while LLMs effectively handle localized tool-enforced failures, they struggle with complex environment and dependency issues.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are a chef running a busy restaurant. You have a complex recipe (your software code) and a strict set of kitchen rules (your Continuous Integration, or CI, system). Every time you tweak a recipe, the kitchen's automated inspector checks it. This inspector doesn't just taste the food; it checks if the stove is on, if the ingredients are fresh, if the chef followed the safety rules, and if the plating looks right.
Sometimes, the inspector rejects the dish. Maybe the salt is wrong, maybe the oven temperature is off, or maybe the recipe calls for an ingredient that isn't in the pantry anymore. Fixing these rejections is hard because the problem might not be the recipe itself, but the kitchen setup or the rules.
The Problem: The "Black Box" of Fixing
Currently, computer programs designed to fix code (Automated Program Repair) are like chefs who only look at the recipe book. They are great at fixing a typo in the ingredients list, but they often fail when the problem is that the oven is broken, the wrong spices were bought, or the kitchen rules changed. Existing tests for these repair programs are too simple; they pretend the kitchen is perfect and only check if the food tastes right, ignoring the messy reality of the whole kitchen.
The Solution: CI-Repair-Bench
The authors of this paper built a new, realistic training ground called CI-Repair-Bench. Think of it as a "Simulation of a Real, Messy Restaurant Kitchen."
- Real Data: Instead of fake problems, they collected 567 real examples of "rejected dishes" from 103 actual software projects on GitHub.
- The Full Inspection: To prove a fix works, the system doesn't just run a taste test. It re-runs the entire kitchen inspection process: checking the stove, the ingredients, the safety rules, and the final taste. If the dish fails any of these checks, the fix is considered a failure.
- The Variety: They categorized the failures into 12 types, ranging from "The plating is messy" (formatting) to "The oven is broken" (environment errors) to "We don't have the right flour" (dependency issues).
The Experiment: Can AI Chefs Fix It?
The researchers tested four different "AI Chefs" (Large Language Models) to see if they could fix these rejected dishes using only the inspector's notes (the error logs).
Here is what they found:
- The "Easy" Fixes: The AI was surprisingly good at fixing "plating" issues. If the error was "your code is not formatted correctly" or "you missed a comma," the AI could fix it about 35% of the time. It's like a chef who is great at wiping the plate clean.
- The "Hard" Fixes: The AI struggled terribly with the complex stuff. When the problem was "the oven is broken" (environment issues) or "we need a specific brand of flour that isn't installed" (dependency issues), the success rate dropped to nearly zero (often less than 9%). The AI couldn't figure out that the problem wasn't the recipe, but the kitchen itself.
- The "Reading the Notes" Factor: The AI's success depended heavily on how well it read the inspector's notes.
- Smart Reading (Agent-based): When the AI was told to carefully read the long, messy notes, summarize them, and think step-by-step, it did much better.
- Keyword Searching (Retrieval-based): When the AI just searched for keywords in the notes (like a simple search bar), it got confused and failed much more often.
- Analogy: It's the difference between a chef who reads the whole complaint letter to understand the context versus a chef who just looks for the word "burnt" and assumes the food is burnt.
The Big Takeaway
The paper concludes that while AI is getting better at fixing small, specific code errors, it is still very bad at understanding the bigger picture of how software runs in the real world.
- Current Limit: AI can fix the "typos" and "style" issues, but it gets lost when the problem involves the environment, dependencies, or complex system configurations.
- The Gap: There is a huge gap between "applying a patch" (changing the code) and "passing the full inspection" (making the whole system work). Most patches the AI made looked correct but failed the full kitchen inspection because they didn't fix the underlying environment or dependency issues.
In short, CI-Repair-Bench is a new, tougher test that shows us exactly where our AI repair tools are strong (fixing small code details) and where they are weak (fixing the complex, real-world systems that run the code). It proves that to fix software in the real world, we need AI that understands the whole kitchen, not just the recipe.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.