Measuring What a Specification Determines: A Formal Semantic-Block Model and an Execution-Judged Benchmark

In the modern software world, a growing number of companies are trying to automate the creation of computer programs using artificial intelligence. Instead of hiring a team of engineers to write every line of code from scratch, they provide a detailed written description of what the software should do, known as a specification, and ask an AI to build it. This approach, called specification-driven development, treats the written plan as the primary instruction manual for the machine. The hope is that if the plan is clear enough, the AI will produce perfect software every time. However, a critical question remains: does a better plan actually make the AI smarter, or do the AI systems already know the answers based on their training? If the AI agrees with the plan simply because it has seen similar plans before, the plan itself isn't doing any real work. This uncertainty makes it difficult to know if a specification is truly high quality or just a document that happens to match what the machine was already going to do.

A team of researchers set out to solve this measurement problem by testing a specific, complex task: moving a massive database from one type of system to another. They created a formal, structured specification for migrating data from an Oracle database to a PostgreSQL database, a process that involves translating thousands of rules about how data is stored and processed. To test if this specification actually helped, they did not just ask the AI to write code and check if it looked right. Instead, they built a rigorous experiment where the same group of AI systems had to perform the migration twice: once with the detailed specification and once without it. The researchers used a live Oracle database and a fresh PostgreSQL system as a strict judge. They ran the code generated by the AI against the original data to see if the results were identical, treating the actual behavior of the software as the only true measure of success.

The study involved three different AI systems acting as independent implementers, all working on the same 75 specific migration tasks. When the researchers compared the results, they found a clear split between what the software could do and how well the AI systems agreed with each other. The specification dramatically improved the ability of the software to run without crashing. Without the specification, only 72 percent of the generated code could be loaded into the new database successfully. With the full specification, that number jumped to 97.3 percent. The plan acted as a guide that helped the AI avoid fatal errors and produce code that actually worked.

However, the story took a different turn when the researchers looked at whether the specification made the AI systems agree more with each other. Before the study, there was a hope that a perfect plan would force all the AI systems to make the exact same decisions, creating a unified solution. The data showed this was not the case. Even without the specification, the AI systems already agreed with each other about 83 percent of the time, likely because they had learned the same standard industry practices during their training. When the specification was added, this agreement rate barely moved, shifting only to 83.8 percent. The plan did not change the AI's mind on the decisions it was already making; it simply helped the AI execute those decisions without breaking.

The researchers also discovered that the way information was presented mattered more than how much information was given. In one experiment, they took a single rule and placed it in different parts of the document. When the rule was buried in a paragraph at the end of a section, the AI followed it only 23 percent of the time. When the same rule was placed in a structured table at the top of the section, compliance rose to 42 percent. Surprisingly, repeating the rule in both places actually lowered the compliance rate to 34 percent, suggesting that redundancy can confuse the system rather than reinforce the instruction. This finding indicates that the structure of the document is more influential than the volume of text.

Perhaps the most significant finding was that a specification can sometimes make things worse. In one specific case, a rule in the specification instructed the AI to use a certain type of data container. The AI followed this rule perfectly, but the result was code that was invalid and would not work. Without the specification, the AI had ignored that specific instruction and used a different, working method on its own. This proved that following a rule does not guarantee a correct outcome, and that a specification can introduce new errors even as it fixes old ones. The study also found that the AI systems had a natural limit to their consistency; when the researchers ran the same test multiple times, the results varied by about 14 percentage points simply due to the random nature of how the AI generates text. This variability meant that small improvements could not be trusted as real progress.

Ultimately, the research concludes that a specification is a powerful tool for making software executable and for helping humans find where the instructions are missing, but it is not a magic wand that forces different AI systems to think alike. The specification successfully reduced the number of broken programs by nearly a quarter, proving its value in ensuring the code runs. Yet, it failed to increase the agreement between different AI systems or to improve the accuracy of the data itself, which remained unchanged at 19 correct results out of 42 tests in both the controlled and uncontrolled groups. The study suggests that for the current generation of AI, a specification acts more like a safety net that prevents catastrophic failure than a guide that elevates the quality of the solution beyond what the AI already knows. The real power of the specification lies not in forcing the AI to agree, but in making the process of building software reliable enough to be checked and verified.