In the vast, intricate landscape of modern software, performance improvements rarely look like new inventions. Instead, they are subtle adjustments to existing codebases, where a single change must fit perfectly with thousands of lines of established logic, strict build rules, and public interfaces. Finding these improvements is difficult because the space of possible changes is enormous, and most attempts fail to build or break the system. To navigate this, researchers have developed automated agents that can write and test code. These agents operate like explorers, but the strategy they use to decide where to go next matters immensely. Some strategies focus entirely on the single best path found so far, piling changes onto it like a climber ascending a single ridge. Others try many different paths at once but start each new attempt from the very beginning, discarding any progress made on previous attempts. A third approach, known as quality-diversity search, attempts to keep a map of many different successful states, preserving variations that are not necessarily the current "best" but might lead to something better later.

This paper introduces a system called LORELEY, which applies this quality-diversity approach to the evolution of entire software repositories. The researchers wanted to know if keeping a diverse archive of past code states, and occasionally returning to them for inspiration, would actually produce better results than simply piling changes onto the current best version or starting fresh every time. They tested this by pitting the LORELEY system against two simpler, more traditional strategies in a controlled experiment using the Zstandard compression library, a critical piece of software used to shrink data files. The goal was to see if the more complex, memory-rich approach could find a superior final version of the code within a fixed budget of attempts.

The experiment was rigorous and matched carefully to ensure a fair comparison. The researchers ran three different search policies on the same frozen starting point of the Zstandard code. The first policy, called Sequential Champion, acted like a relentless climber: it took the best version found so far and asked the agent to improve it further, discarding all other branches. The second, Independent Root, was like a group of hikers starting from the base camp every time; each attempt began from the original code, ignoring any improvements found by others. The third, LORELEY, maintained an archive of many different valid code states. When it needed to generate a new idea, it could pick a base from this archive and also look at other stored states for inspiration, hoping that combining a less obvious starting point with a fresh idea would yield a breakthrough.

The study ran for a specific budget of forty-eight attempts, or "jobs," for each policy. In the world of automated coding, a job is a complete cycle where the system picks a starting code version, an agent writes changes in an isolated environment, and an external tester builds and measures the result. The researchers measured the final performance of the best code found by each policy using a separate set of data that the agents had never seen during their search. This "holdout" test ensured that the results were genuine improvements and not just lucky guesses that worked only on the training data.

The results showed that the Sequential Champion strategy, which simply kept building on the best version, had the highest observed mean and median performance after forty-eight jobs. The LORELEY system, despite its complex archive and ability to revisit old ideas, finished slightly behind the champion. The Independent Root strategy, which never remembered past successes, performed the worst. However, the data did not establish a statistical advantage for the Quality-Diversity (QD) approach over either control; the confidence intervals included zero, meaning the experiment could not confirm that QD improves final held-out performance over the simpler strategies, nor could it establish equivalence. While LORELEY did successfully keep a diverse set of code states in its archive and did occasionally sample from them, this behavior did not translate into a statistically proven better final outcome within the time limit of the experiment. The system did not demonstrate that keeping a map of many paths is definitively better than focusing on the single best path for this specific task.

However, the story is not entirely one of failure for the complex approach. The researchers observed that the LORELEY system did engage with its intended mechanism. It successfully retained code states that were not the current best, and it did sample these non-champion states later to use as a base or as inspiration. In four out of seven test runs, the final winning code from the LORELEY system had ancestors in its history that were not the current leaders at the time they were added to the archive. This proved that the system could hold onto and revisit "stepping stones"—intermediate ideas that might not be perfect on their own but could lead somewhere new. Yet, in this specific experiment with a limited number of attempts, these stepping stones did not help the system beat the simpler, more direct strategy in a statistically significant way.

The paper also looked at earlier, smaller campaigns where the system was used on different software libraries, including a Python library for handling text and a separate revision of the compression tool. In these cases, the system successfully produced significant improvements, such as a nearly seven percent speedup in one library and a twenty-five percent gain in another. These successes demonstrate that the system is capable of finding complex, multi-file improvements when given the right conditions. But the controlled comparison with the simpler strategies showed that, at least for the Zstandard task with a budget of forty-eight jobs, the extra complexity of maintaining a diverse archive did not provide a statistically established advantage over simply focusing on the current best version.

Ultimately, the study offers a nuanced view of automated software evolution. It confirms that a system can be designed to remember and reuse a wide variety of past states, and that it can successfully navigate a complex codebase to find improvements. But it also suggests that for certain tasks and within specific time limits, the most effective strategy may be the straightforward one: find the best thing you have, and keep making it better, rather than trying to manage a sprawling map of possibilities. The researchers did not find that the complex method was useless, but they did find that it did not win this particular race with statistical significance. The results remain specific to the tools and constraints used, leaving open the question of whether a longer search or a different type of problem might eventually favor the diverse, memory-rich approach.