Cloud-Native Architecture and Pipeline Optimization for Built-Environment Data

In the modern world of software, building a program is no longer a solitary act performed on a single computer. It has become a massive, industrial operation where code is constantly assembled, tested, and shipped out to users around the clock. This process generates a vast ocean of digital byproducts: the finished software files, the records of what was changed, and the logs showing how long each step took. Researchers call this the "built-environment data." As companies grow from dozens of engineers to thousands, the volume of this data swells, often overwhelming the traditional systems designed to manage it. These older systems were built for static, unchanging environments, much like a factory with a fixed number of machines that never sleep. When the demand spikes, these factories clog up, causing delays and wasting money. To solve this, experts have turned to a new way of organizing these digital factories, one that treats software components like portable containers that can be instantly created, moved, and discarded as needed. This approach, known as cloud-native architecture, promises to make the entire process flexible and efficient, but it requires a new kind of management to truly work.

Jim Dorsey, a researcher at Miami University of Science and Technology, set out to investigate how to best combine this flexible, container-based architecture with specific strategies to speed up the building process. The core question was whether simply moving to a modern, cloud-based system was enough, or if it needed to be paired with clever tricks to manage the flow of work. The study focused on the "pipeline," the sequence of steps that turns raw code into a finished product. Dorsey proposed a framework that uses these portable containers to handle the work, while simultaneously applying techniques like running independent tasks at the same time, saving the results of expensive steps to reuse them later, and dynamically adjusting the amount of computing power available based on how busy the system is. The goal was to see if this combination could reduce the time it takes to build software, lower the cost of running the systems, and make the whole process more reliable.

To test this idea, the researcher built a controlled experiment using a simulated environment that mimicked a real-world cloud setup. They created three different versions of a software building system to compare against each other. The first was a traditional setup, representing the older, static way of doing things where computers are always running whether they are needed or not. The second version used the modern cloud-native approach, where computers are only started when there is work to do and shut down immediately after. The third version took that modern approach and added the extra optimization strategies, such as the smart scheduling and caching mentioned earlier. They ran a series of realistic software projects through each of these three systems, repeating the process dozens of times to gather accurate data on how long each build took and how much it cost.

The results showed a clear progression in performance. The traditional system took an average of 18.4 minutes to complete a build. When the researchers switched to the modern cloud-native system without any extra tricks, the time dropped to 12.7 minutes, a significant improvement of about 31 percent. However, the biggest leap came from the fully optimized system. By adding the strategies for parallel work and smart resource use, the average build time fell further to just 7.2 minutes. This represented a total reduction of nearly 61 percent compared to the old way of doing things. The speed increase was not just a matter of time; it also meant the system could complete more builds per hour, jumping from about 3.3 builds an hour in the old system to 8.3 in the fully optimized one.

Beyond speed, the study looked at how efficiently the system used its resources and how much it cost to run. In the traditional setup, the computers were often sitting idle, waiting for work, which resulted in a very low usage rate of about 23 percent. This wasted energy and money. The modern cloud-native system improved this to 47 percent, and the fully optimized version reached 62 percent, meaning the computers were working much harder when they were running. This efficiency translated directly into savings. The cost to build a single piece of software dropped from $4.82 in the traditional system to $3.15 in the modern system, and finally down to $1.87 in the optimized system. This was a reduction of more than 60 percent in cost, proving that the combination of the new architecture and the optimization strategies was not just faster, but also much cheaper.

Reliability and the ability to see what was happening inside the system also improved. The older system had a success rate of about 94 percent, meaning some builds failed. The optimized system raised this to 98.4 percent, with fewer errors caused by running out of computing power. Furthermore, the ability to track and understand the system's performance, known as observability, jumped from covering 68 percent of the process in the old system to 97 percent in the optimized one. This means that when something went wrong, it was much easier to find the cause. However, the study noted that these gains did not come without a price. The most optimized system was the most complex to set up and manage. It required more careful planning and deeper technical knowledge to maintain. The researchers found that while the benefits were substantial, organizations had to weigh the desire for speed and low cost against the increased complexity of managing such a sophisticated system.

The study concluded that simply adopting a modern, cloud-based architecture is a necessary first step, but it is not enough on its own to solve the problems of speed and cost. The architecture provides the flexible foundation, but it is the specific optimization strategies—like running tasks in parallel and reusing previous work—that unlock the full potential. The research suggests that for organizations looking to modernize, the best path is to first build a system that can be observed and measured, and then layer on these optimization techniques. While the most advanced setup offers the best performance, the study indicates that a middle ground is often sufficient, offering significant improvements over the old ways without the extreme complexity of the fully optimized version. Ultimately, the work provides a clear roadmap for how to manage the growing flood of digital data in software development, turning a potential bottleneck into a streamlined, efficient engine.