BUILD-AND-FIND: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases
This paper introduces BUILD-AND-FIND, an effort-aware evaluation protocol that assesses the quality of agent-generated codebases by measuring not only their behavioral correctness but also the accuracy and inspection effort required for downstream agents to recover the original design intent and specifications.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
The Big Idea: It's Not Just About the Answer, It's About the Note
Imagine you hire a chef (an AI agent) to cook a complex meal based on a secret recipe you gave them.
- Old Way: You taste the food. If it tastes good, the chef gets a pass.
- New Problem: What if the chef cooks a delicious meal, but they hide the secret ingredients in a locked box, write the recipe in invisible ink, or leave the kitchen in a mess? Later, another chef (a different AI) needs to come in, taste the food, and figure out how it was made so they can fix a burnt edge or add more salt. If the first chef didn't leave clear clues, the second chef might fail, even though the food tasted great.
This paper introduces a new way to test AI coders called BUILD-AND-FIND. Instead of just checking if the code "works" (tastes good), it checks if the code is easy to read and understand for the next person (or AI) who has to work with it.
The Two Roles: The Builder and The Finder
The researchers set up a game with two distinct roles:
The Builder (The Chef):
- They are given a hidden "specification" (the secret recipe).
- Their job is to build a complete software project (the kitchen and the meal) based on those instructions.
- They don't know anyone else will be looking at their work immediately; they just have to make it work.
The Finder (The Inspector):
- They are given only the finished code (the kitchen and the meal). They are not allowed to see the original secret recipe.
- They are given a list of multiple-choice questions about the design choices (e.g., "Why did the chef choose this specific type of oven?" or "Where is the salt stored?").
- Their job is to look through the code, find the evidence, and answer the questions correctly.
The Scorecard: Accuracy vs. Effort
The paper measures two main things:
Did they get it right? (Accuracy)
- Can the Finder figure out the correct answer just by looking at the code?
- Analogy: Did the Inspector successfully find the hidden spice jar?
How hard did they have to look? (Inspection Effort)
- This is the paper's big innovation. If two Builders both make code that allows the Finder to get the right answer, which one was easier to figure out?
- The researchers measure "effort" by counting how many bytes of code the Finder had to read or search through.
- Analogy: If Chef A left the salt jar on the counter, and Chef B hid it inside a locked safe that required a complex code to open, both chefs made edible food. But Chef A made it easier for the next person to work with. The paper rewards the chef who leaves the "salt" on the counter.
Why This Matters
The paper argues that in the future, AI agents will be working in teams. One AI writes a program, and another AI has to fix or update it later.
- If the first AI writes code that is "correct" but messy or confusing, the second AI will struggle.
- BUILD-AND-FIND proves that we can measure how "legible" or "communicative" an AI's code is, separate from whether the code actually runs.
The Results (What They Found)
The researchers tested this with several powerful AI models (like GPT-5, Claude Opus, and others) on two types of tasks: building a mini-database and building a tiny web server.
- The "High Prior" Problem: The questions they asked were things that experienced engineers usually know by heart (like "databases usually save logs before writing"). Because the AIs are smart, they could guess the answers correctly just by using their general knowledge, even without reading the code.
- The Solution: Because everyone got the answers right, the researchers couldn't just count "correct answers." Instead, they looked at how much reading the AIs had to do.
- Some AI models wrote code where the answers were obvious and easy to find (low effort).
- Others wrote code where the answers were buried deep in the files, requiring the Finder to read a lot more to find them (high effort).
- The Winner: The models that produced code requiring the least amount of reading to find the answers were considered the best at "communicating" their design choices.
The Safety Net (Controls)
To make sure the AIs weren't just cheating or guessing, the researchers added safety checks:
- Question-Only Test: They asked the Finder the questions without showing them any code. If the AI got it right here, it was just guessing based on general knowledge.
- Spec-Only Test: They showed the Finder the secret recipe but no code. This proved the recipe was clear.
- The "Gate": The researchers only counted the "effort" score if the Finder actually got the answer right. If the Finder couldn't find the answer at all, the code was considered a failure, regardless of how little they had to read.
Summary
BUILD-AND-FIND is a new test for AI coders. It asks: "If you write this code, will another AI be able to easily understand why you made the choices you did?"
It moves beyond asking "Does the code run?" to asking "Is the code a good communication tool for the future?" The best AI isn't just the one that builds the right thing; it's the one that builds the thing in a way that makes it easy for the next person to pick up and continue working.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.