Specification-first convergence with an AI coding agent: a case study of dismantling a core architectural invariant across 189 files in a 717k-line codebase with no test oracle and no human code review

Imagine you are trying to teach a super-smart robot how to build a house. Usually, the robot is great at laying a single brick or painting one wall, but if you ask it to redesign the entire foundation while the house is already being lived in, it tends to get confused. It might knock down a load-bearing wall by accident, or forget that the kitchen needs to stay connected to the dining room. In the world of computer science, this is called "refactoring" a massive codebase. The big question researchers are asking right now is: Can an AI agent do this huge, dangerous job all by itself, without a human boss checking every single line of code?

To understand the challenge, think of a computer program like a giant, intricate clockwork machine. Inside, there are "invariants"—these are the unbreakable rules that keep the gears turning, like "the main spring must always be wound" or "the hands must never jump backward." Most AI coding tools are like apprentices who are great at fixing a loose screw but terrible at rewiring the whole clock without breaking the timekeeping. They usually need a human to look over their shoulder and say, "Wait, you can't do that!" But humans get tired, and they can't hold the entire map of a giant clock in their heads at once. This paper explores a new way of working: instead of checking the robot's work after it's done, we force the robot to write a perfect, detailed instruction manual before it touches a single gear. Then, we have the robot check its own work against that manual, over and over, until the manual and the machine match perfectly.

The Story of the "Ghost" Panel

This paper tells the story of one specific, high-stakes experiment where a single AI agent was asked to perform a "magic trick" on a massive software program. The program was a 717,725-line application (imagine a library with nearly 720,000 pages of instructions) that acted as an AI coding assistant. The task was to break a fundamental rule the software had followed since day one: the rule that said, "If you close the window showing the AI's work, the AI must stop thinking."

The goal was to change this so that if you closed the window, the AI would keep working in the background, like a ghost. When you opened the window again, the AI would instantly "re-attach" to the conversation, picking up exactly where it left off without losing a single word or repeating itself. The author of the paper, a human developer, believed this was so complex and risky that the only safe way to do it would be to throw away the old code and rewrite the whole thing from scratch.

The "Specification-First" Protocol

Instead of rewriting everything, the author used a method called "Specification-First Convergence." Think of it like a game of "Telephone" played with a very strict referee, but the referee is the AI itself.

1. The Plan: The human gave the AI a simple request in plain English: "Make the AI keep working even when the window is closed."
2. The Blueprint: The AI didn't start coding immediately. Instead, it wrote a massive, 55-page formal "specification"—a detailed blueprint of exactly how the change would work, down to the smallest detail.
3. The Audit (Refinement): This is where the magic happened. The AI was asked to compare its own blueprint against the real, existing code 14 times. In each round, it found mistakes in its own plan. "Oh, I forgot that the background process needs a new button to stop it," or "I didn't account for what happens if the window closes while the AI is thinking." It rewrote the blueprint 14 times, fixing about 85 errors in the plan before writing a single line of new code.
4. The Build: Once the blueprint was "frozen" (meaning it was perfect and wouldn't change anymore), the AI started coding. It didn't just write files; it created patches (small updates) to 189 different files.
5. The Double-Check (Verification): This was the most critical part. The AI was asked to read its own new code and compare it to the frozen blueprint 17 times. It found 116 more errors—tiny architectural glitches or logic holes—and fixed them. It kept doing this until two rounds in a row found zero mistakes.

The Results: A Ghost That Doesn't Haunt

The entire process took three days and cost $2,430 in computer processing fees. The result? The AI successfully dismantled a core rule of the software across 189 files without a human ever looking at the generated code.

When the human finally ran the program for the first time, it worked perfectly. You could close the window, watch the AI keep working in the background, and reopen the window to see the conversation resume instantly, with no lost words and no duplicates. The software behaved exactly as the blueprint promised. The author noted that the code was so clean that later, when a small interface tweak was needed, it only required changing one single file, suggesting the AI had done a good job of keeping the structure organized.

What This Means (and What It Doesn't)

This paper suggests that for very difficult, high-risk software changes, we might not need to rely on human reviewers to catch every mistake. Instead, we can use the AI to rigorously check its own plans and its own code against a frozen standard, over and over, until it converges on perfection.

However, the paper is very careful about what it claims. It admits this was just one specific task on one specific codebase. It doesn't prove that this will work for every AI or every type of software problem. It also notes that because the code is private, no one else can run the exact same experiment to see if they get the same result. But as a proof of concept, it shows that an AI, when forced to plan meticulously and audit itself repeatedly, can perform a "surgical" operation on a giant software system that humans usually think is too dangerous to touch without a full rewrite.