Specification-first convergence with an AI coding agent: a case study of dismantling a core architectural invariant across 189 files in a 717k-line codebase with no test oracle and no human code review

In the world of software, programs are built like vast, intricate cities. They consist of millions of lines of instructions that tell a computer how to behave, how to remember things, and how to react when a user clicks a button. For decades, the standard way to change these cities has been to send a human architect to inspect every new brick an artificial intelligence might lay. This human review is considered essential because the connections between different parts of a program are so deep and tangled that a small mistake in one corner can cause a collapse in another. When a change is large enough to touch hundreds of files at once, the task of checking the work becomes a bottleneck; no single human can hold the entire map of the changes in their mind at once. This limitation has led many to believe that for the most complex architectural shifts, the only safe path is to tear the old structure down and build a new one from scratch, a process that is slow, expensive, and risky.

A recent case study challenges the idea that human review is the only way to ensure safety in such massive changes. It explores a method where the focus shifts from checking the finished product to perfecting the blueprint before a single brick is laid. Instead of asking an artificial intelligence to write code and then hoping a human can find the errors, this approach asks the machine to first write a detailed, formal description of exactly what it intends to do. This description is then rigorously checked against the existing code, not to see if the code works, but to see if the plan itself makes sense. The plan is refined, corrected, and frozen until it stops producing findings. Only then does the machine begin to build, and even then, it is constantly audited against that frozen plan. The result is a demonstration that a massive, complex architectural change can be completed without a single human ever reading the generated code, provided the process of defining and verifying the intent is robust enough.

The experiment took place within a large, proprietary software application used for artificial intelligence coding assistance. The system, written in a language called TypeScript, contained over 717,000 lines of code spread across nearly 3,650 files. It was a living, breathing piece of software used daily, not a theoretical model. The task assigned to the artificial intelligence agent was to dismantle a core rule that had governed the system since its creation. This rule, known as an invariant, guaranteed that whenever a user started a conversation with the AI, the window displaying that conversation would remain open for the entire duration of the request. If the user closed the window, the conversation would die. The goal was to break this rule: the user should be able to close the window, and the conversation would continue running in the background, ready to be reopened and resumed exactly where it left off, without losing any data or repeating any words.

This specific change was considered nearly impossible to achieve through standard refactoring. In the author's assessment, the interdependence of the code was so tight that modifying the lifetime of a conversation without breaking the entire system would normally require a complete rewrite of the affected components. The task involved managing complex timing issues, such as what happens when a window closes while data is still streaming in, and how to reattach a user to a live stream that has been running without them. To solve this, the author employed a five-phase protocol designed to keep the artificial intelligence on a strict track. The process began with the agent analyzing the request and producing a formal specification, a detailed document describing exactly how the change should work.

The most critical part of the process was the refinement phase. The agent was asked to take its own specification and check it against the real source code, looking for any contradictions or missing details. This was not a one-time check. The agent ran through fourteen cycles of this audit. In each cycle, it found errors in its own plan—perhaps a file it had forgotten to update, or a dependency it had misunderstood—and rewrote the specification to fix them. Over these fourteen rounds, the plan was corrected approximately eighty-five times, expanding the scope of the change from 110 files to 160 files as hidden connections were discovered. By the end of the fourteenth cycle, the specification was frozen. It was a fixed blueprint that had been audited against the reality of the codebase until the final cycle returned no findings, at which point it was locked as the reference for all subsequent phases.

With the plan locked in place, the agent moved to the implementation phase. It was instructed to generate the code changes required to match the frozen specification. The agent refused to make partial changes, correctly identifying that a half-finished migration would break the system. Instead, it executed the work in three distinct, confirmed steps. Once the code was written, the verification phase began. Just as the plan had been checked against the code, the new code was now checked against the frozen plan. The agent ran seventeen cycles of this audit, comparing the actual code on the screen to the written rules in the specification. In each cycle, it found and corrected deviations, fixing subtle architectural flaws that a human might have missed. Across these seventeen rounds, the agent corrected 116 defects in the code itself. The process stopped only when two consecutive audits returned zero errors, confirming that the code matched the plan within the defined convergence criterion.

The entire operation touched 189 files, with a total of 288 files changed when including the extraction of the old code. The changes involved over 34,000 new lines of code and the deletion of more than 16,000 lines. Remarkably, this massive restructuring was completed in three days. The cost of the artificial intelligence processing required for the task was 2,430 US dollars. Throughout the entire three-day period, the program was never run once by a human. The first time the software was actually executed was after the seventeenth verification cycle, when the author finally tested the new behavior.

The result was a success. When the program was launched, the new behavior worked exactly as the specification had described. A user could start a conversation, close the window, and the conversation would continue running in the background. When the user reopened the window, the conversation resumed instantly, with no loss of data and no duplication of text. A new stop button appeared in the sidebar, allowing the user to kill the background process if needed. The existing automated tests, which had been running on the software for years, showed no failures, indicating that the new system did not break any of the old functionality. The software was released to the public as version 2.3.0, and in the roughly thirty sessions of use that followed the release, no bugs were observed.

This case study does not claim that artificial intelligence can now replace human engineers in all scenarios, nor does it suggest that this method works for every type of problem. The author is careful to note that this was a single, specific instance on a single codebase, and the results cannot be automatically assumed to apply to other systems or other tasks. The study explicitly rules out the idea that a pre-existing set of tests could have solved this problem, because the desired behavior did not exist before the change was made. There was no "correct" answer to check against in the old code; the correctness had to be defined from scratch. The study also acknowledges that the process relied on a specific, powerful model of artificial intelligence and that the results might differ with weaker models.

The significance of the work lies in its demonstration of a new way to manage complexity. By shifting the burden of quality control from inspecting the final product to rigorously defining and verifying the plan, the researchers showed that a machine could navigate a minefield of interdependent code without human intervention. The key was not that the machine was infallible, but that the process allowed it to catch its own mistakes repeatedly before they became permanent. The specification was challenged fourteen times, and the code was checked seventeen times, creating a loop of self-correction that drove the system toward a match between intent and reality based on an empirical stopping rule. The logs of this entire process, spanning over 1,500 pages, have been published for anyone to inspect, offering a transparent look at how a machine can dismantle a core architectural rule and rebuild it, brick by brick, without ever needing a human to look over its shoulder.