Orchestrating human–AI software delivery: a retrospective longitudinal field study of three software modernization programs

In the world of building software, the work has long been understood as a human craft. Teams of engineers gather to understand a problem, sketch out a plan, write the code that makes a program run, and then spend considerable time checking that code for mistakes before releasing it to the public. For decades, the measure of success in this field was how quickly a human could complete these steps. Recently, a new kind of helper has entered the workshop: artificial intelligence. These systems can read vast amounts of code and suggest new lines of text in seconds. Early reports suggested that simply handing these tools to individual programmers would make them faster, much like giving a carpenter a power saw instead of a hand saw. But building a large software system is not just about cutting wood faster; it is about ensuring every piece fits together, that the structure is sound, and that the final product works as intended. The critical question for the industry has become whether these intelligent helpers can truly improve the entire process of building a system, or if they merely speed up the initial steps while creating new problems later on.

A team of researchers set out to answer this question by watching how a real company used these tools over time. They did not run a controlled experiment in a lab where they could isolate a single variable. Instead, they looked back at three major software projects that the company had already completed. These were significant undertakings: one involved moving a banking system from an old language called COBOL to a modern one, another was a massive update to a large accounting platform, and the third was a modernization of a mortgage application. The researchers examined how these projects were delivered under five different setups. They started with a traditional approach, where humans did all the work, and then moved through four successive versions of a new platform that combined human engineers with artificial intelligence agents.

The first version of this new platform was essentially a collection of tools. In this setup, the artificial intelligence agents acted as assistants to individual engineers. They could analyze old code, write documentation, and even execute specific tasks on their own. The results here were mixed. The team found that the early stages of the projects, such as understanding the old code and planning the new one, became much faster. However, this speed came at a cost. Because the workflow lacked a central system to check the work as it progressed, the quality of the final output suffered. More errors slipped through to the later stages of testing, and the team had to spend more time fixing them. It was as if the builders had learned to lay bricks twice as fast, but they were laying them in the wrong places, requiring a great deal of time to tear them down and start over.

The story changed as the researchers moved to the later versions of the platform. The key difference was not just that the artificial intelligence got smarter, but that the way humans and machines worked together was completely reorganized. In the most advanced setup, the artificial intelligence was no longer just a tool for individuals; it became part of a coordinated team. The system created a shared workspace where the plan, the code, and the review process were all connected. The artificial intelligence agents could now check their own work against specific rules before a human ever saw it, and they could submit their code for review in the same way a human would. This version also introduced a hybrid approach where humans and machines worked side-by-side on the same tasks, sharing responsibility.

When the researchers compared the final, most advanced setup to the traditional human-only approach, the results were striking. The total time it took to complete the three projects dropped from thirty-six weeks down to just over nine weeks. This was not just a matter of working faster; the quality of the work also improved. The number of errors found during the final testing phase fell dramatically, from more than eight issues for every hundred tasks down to just two. Furthermore, the percentage of requirements that were met correctly on the very first release rose from seventy-seven percent to over ninety percent. The study also modeled the effort required, estimating that the work of the new team, which included artificial intelligence agents, was equivalent to eighty-seven percent less effort from senior-level human experts compared to the old way of working.

The researchers were careful to note that these findings describe what happened in this specific company with these specific projects. They did not claim that the artificial intelligence alone caused these results. Instead, they suggested that the magic lay in the orchestration. The early versions, which simply gave tools to individuals, improved speed but hurt quality. The later versions, which integrated the artificial intelligence into a structured workflow with shared context and rigorous checks, improved both speed and quality together. The study implies that the future of software delivery does not lie in replacing humans with machines, nor in simply giving machines to humans to use as they please. Rather, the greatest gains come when humans and machines are woven together into a single, coordinated system where the strengths of one compensate for the weaknesses of the other.

This research offers a clear picture of how artificial intelligence is maturing in the real world. It moves beyond the simple idea of a faster typing assistant to show how a team of humans and machines can operate as a unified whole. The data suggests that when artificial intelligence is embedded in a workflow that includes planning, validation, and review, it can help teams deliver complex systems much faster and with fewer mistakes. While the study is limited to one organization and cannot prove that this will happen everywhere, it provides a strong, descriptive record of what is possible when the technology is treated not just as a tool, but as a partner in a carefully designed process. The path forward, the evidence suggests, is not about how fast the machine can write code, but about how well the team can organize the work so that the machine and the human can build something better together.