NumaRing: Topology-Aware Routing for NUMA-Local MPMC Queues, and What Broke When We Optimized It

Modern computers are built like bustling cities with multiple districts, each housing its own processing power and memory. When a program needs to do work, it sends a request to a specific district. If the data it needs is already in that district's local memory, the task happens instantly. But if the request must travel to a different district to fetch information, the journey takes significantly longer. This delay, caused by the physical distance between the districts, is a fundamental limit of how these machines are built. For decades, software engineers have tried to write programs that keep data and the workers who use it in the same district, hoping to avoid the slow cross-country trips. The challenge is that when many workers are trying to access a shared list of tasks at the same time, the traffic jams they create can be just as damaging as the distance itself.

A researcher set out to build a better way to manage these shared lists, specifically for computers with two distinct districts. They created a system called NumaRing, designed to keep workers and their data local to their own district whenever possible. The idea was simple: if a worker is in the first district, it should only look at the list in the first district. If that list gets full or empty, the system would then move a batch of tasks to the other district in one go, rather than moving them one by one. This approach promised to keep the fast, local traffic flowing while minimizing the slow, long-distance trips. However, when the researcher put their system to the test, they discovered that their best intentions had hidden traps. By measuring the system with extreme precision rather than guessing how it worked, they found that two specific mistakes were slowing it down more than the hardware itself, and that a common piece of advice for fixing computer slowdowns was actually making things worse.

The researcher began by building their system on a cloud computer with two districts, each containing sixteen virtual processors. They filled it with a steady stream of tasks, watching how long it took for a task to go from the start of the line to the end. In the beginning, the system was surprisingly slow. The researcher realized that every single time a worker tried to add or remove a task, the software asked a question: "Which district am I in right now?" This question seemed harmless, but the answer took a long time to compute. The software was re-calculating the location from scratch every single time, even though the worker's location rarely changed. This repeated calculation was like a driver stopping at every intersection to ask for directions, even though they knew exactly where they were going. The cost of this question was so high that it took up more than eleven times the effort of the actual task of moving the data.

Once the researcher fixed this by remembering the location and only checking it when necessary, the system sped up dramatically. The number of tasks processed per second jumped six to seven times higher. But the story did not end there. When they added more workers to the machine, the system hit a new wall. The workers were still waiting too long, especially when the system was under heavy pressure. Digging deeper, they found a second problem in the way workers shared tasks between districts. When a worker needed to grab a batch of tasks from another district, every single worker was fighting over the same small counter to decide who would go next. This created a massive traffic jam at the gate. By giving each worker their own private counter to use, the researcher removed this bottleneck. This change was even more dramatic, cutting the time a worker had to wait in the middle of the line by more than two hundred times.

With these two major fixes in place, the researcher expected their system to be a champion. They had eliminated the software errors that were holding it back. However, when they pushed the machine to its absolute limit with thirty-two workers, the system still could not reach the speed they had originally hoped for. The researcher then tested a standard technique used to solve computer slowdowns called "backoff." The idea behind backoff is that if a worker fails to grab a task, it should wait a tiny moment before trying again, hoping the line will clear up. In many situations, this pause helps. But in this specific, high-pressure environment, the pause was a mistake. The researcher measured that waiting actually cost them fifteen to thirty percent of their total speed. The fastest path was to keep trying immediately, because the hardware was already handling the conflict efficiently enough that waiting only wasted time.

The final picture that emerged was one of both success and hard limits. The researcher successfully built a system that kept data local and fixed two major software bugs that were causing massive delays. They proved that a common optimization strategy could be harmful in certain high-speed scenarios. Yet, despite these victories, the system still could not process tasks as fast as the original design goal. The researcher concluded that the remaining slowdown was not a software error they could fix, but a physical limit of the machine itself. The distance between the two districts and the bandwidth of the road connecting them created a ceiling that no amount of clever coding could break through with this design. They reported their findings honestly, showing exactly where their system succeeded, where it failed, and why the hardware itself was the final judge. Their work serves as a reminder that in the world of high-speed computing, understanding the physical machine is just as important as writing the code.