Protocol-Governed Human–AI Software Engineering: Autonomy Without Authority
This paper presents and empirically evaluates a Protocol-Governed Computing architecture that separates activity autonomy from authorization authority, demonstrating how human oversight can govern AI-driven software engineering through explicit admission, promotion, and sealing protocols while identifying specific areas where guard mechanisms require refinement.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the modern world of software creation, a new kind of worker has arrived: an artificial intelligence capable of writing code, designing systems, and fixing errors without ever needing to sleep. This shift has transformed how programs are built, but it has also created a quiet, dangerous confusion about who is actually in charge. For decades, the person writing the code was also the person deciding what that code was allowed to do. When a human developer merged a change, they were implicitly saying, "I have checked this, and it is safe to run." But when an AI writes the code, the human often only reviews the final product. This creates a gap: the AI has the power to build, but the human still holds the power to authorize. The question is whether these two powers can be separated. Can a machine be allowed to do all the heavy lifting of engineering while a human, or a strict set of rules, remains the only one who can decide what the final software is permitted to do?
This is the central puzzle tackled by a recent study from an independent researcher, Bhash Ganti. The work explores a specific architectural idea called Protocol-Governed Computing. In simple terms, this is a system where the rules of what a program can do are locked down before the program is even built. Instead of checking the code after it is written to see if it follows the rules, the system forces the builder to construct the code inside a cage of pre-approved permissions. The builder can design and assemble the parts, but they cannot change the cage itself, nor can they decide to let the finished machine run if it hasn't passed a specific, separate check. The goal was to see if an artificial intelligence could perform the entire job of software engineering—analyzing needs, designing solutions, and writing the code—without ever gaining the authority to change the rules or force the software to run.
To test this, the researcher set up a controlled environment where an AI agent was given a specific task: to add a new feature to an existing software system. The agent was not just a simple assistant typing out suggestions; it was an autonomous worker that had to navigate a complex, nine-step process. This process began with a human defining the scope of the project and the rules that would govern it. The AI then had to analyze the current system, design a new feature, and write the necessary code. Crucially, the AI could not simply write the code and save it. Every step it took was recorded, and every piece of work it produced had to pass through a series of automated checkpoints. These checkpoints acted as a gatekeeper, verifying that the new work matched the original rules and did not try to sneak in unauthorized changes. If the AI tried to break the rules, the system would stop it. If it followed the rules, the work was sealed into a new, immutable state that could be run by the software.
The experiment was successful in showing that the AI could indeed do the work. The agent successfully analyzed the existing system, designed a new lending feature for a digital library, and produced all the necessary code and documentation. It did this by querying the existing system to understand its current state, rather than guessing or remembering details from its training. It produced a candidate for the new software version that was fully compliant with the pre-set rules. However, the most important finding was not that the AI could build, but that it could not authorize. The system was designed so that the AI had no way to promote its own work to become the official, running version of the software. Even though the AI built the candidate, a separate human operator had to explicitly approve it for it to become the new baseline. The AI had the autonomy to construct, but it had zero authority to decide what the system would do next.
The study also looked closely at the evidence to ensure the system was actually working as intended. The researchers tested whether the safety mechanisms were real or just theoretical. They tried to trick the system by removing certain safety checks to see if the AI could slip through. In four out of six specific tests, the system correctly caught the violations and refused to proceed. However, in two other tests, the system failed to detect that a safety check had been removed, even though the check was supposed to be there. This was a significant discovery: it showed that while the architecture was sound, the specific tests used to prove it were sometimes too weak. A passing test did not always mean the safety guard was actually doing its job. This highlighted a critical lesson for the future: having a rule in place is not the same as having proof that the rule is being enforced.
Ultimately, the research demonstrates that it is possible to give an artificial intelligence the freedom to engineer complex software without giving it the power to dictate the rules of that software. The separation of "doing the work" from "deciding what is allowed" is a structural feature of the system, not just a policy. The AI can be the builder, but the authority remains with the sealed rules and the human operator who approves the final step. This does not mean the AI is safe because a human is watching; it means the system is designed so that the human's role is to set the boundaries and approve the result, while the AI operates entirely within those boundaries. The study concludes that we can increase the autonomy of our digital workers without transferring the authority over our digital systems to them, provided we build the right kind of cage around the work. The challenge that remains is ensuring that the tests we use to verify these cages are strong enough to catch every attempt to break them.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.