In the quiet hum of a personal computer, a new kind of intelligence is taking root. Large language models, the engines behind modern chatbots and writing assistants, are increasingly moving from massive data centers to local machines like desktops and workstations. This shift allows users to run these powerful tools without relying on the internet, but it introduces a physical constraint: the computer's memory is limited. To make these models fit, researchers use a technique called "mixture of experts." Imagine a vast library of knowledge where, for every question asked, only a few specific books are pulled off the shelf to be read, while the rest remain untouched. This keeps the work fast and efficient. However, on a personal device, even the "few books" needed for a single moment might not all fit in the computer's active memory at once. When the computer needs a book that isn't there, it must pause its thinking to fetch it from a slower storage drive, leaving the processor idle and waiting.

For years, the standard solution to this waiting game has been prediction. The system tries to guess which books will be needed next and fetches them in advance while the processor is working on the current task. This is like a librarian who, while you are reading one chapter, quietly pulls the next likely chapters off the shelf so they are ready when you turn the page. But a team of researchers from Seoul National University and Yale University has discovered that this strategy often fails when the computer already keeps a small cache of books in its active memory. As that cache fills up with the most useful books, the librarian's guesses become less accurate, wasting time and bandwidth fetching books that will never be opened. Instead of trying to guess the future, the researchers found a better way: simply let the work happen in a different order. By allowing different requests to move forward at their own pace, the system can reveal exactly what is needed next without guessing, turning the idle waiting time into productive work.

The researchers, led by Hongseung Yu and Kyunghan Lee, built a new system called Bevy to test this idea. They observed that when multiple requests are sent to the model at the same time—such as a user asking for a code review while another part of the software checks an email—the requests do not all need the same books at the same time. In traditional systems, the computer waits until every single request has gathered all its necessary books before it starts the next step for any of them. This creates a bottleneck where the processor sits still while the storage drive works. Bevy changes this rhythm. It allows a request that has all its books ready to move ahead to the next step immediately, while other requests wait for their missing books to arrive. As the ready request moves forward, it reveals exactly which books it will need for the next step. The system then fetches those specific books while the processor is busy with the ready request. This creates a seamless overlap where the storage drive and the processor are both working at the same time, but without any need for prediction.

The key to making this work smoothly is a self-balancing scheduler that decides how to group the requests. If the system simply let every request run alone, the processor would have to read the same shared books from memory over and over again for each request, slowing everything down. If it grouped them all together too rigidly, it would lose the benefit of letting the fast requests move ahead. Bevy solves this by grouping requests dynamically, layer by layer. If many requests are ready at the same time, it batches them together to maximize efficiency. If only a few are ready because others are still waiting for books, it lets those few run ahead in smaller groups. This flexibility ensures that the processor is never idle waiting for a group to form, and the storage drive is never idle waiting for a request to finish. The system adapts in real-time to the changing needs of the requests, creating a flow where computation and fetching happen in parallel naturally.

The results of this approach were measured on two different local computing platforms, one using an Apple chip and the other an NVIDIA Jetson AGX Thor Developer Kit with 128 GB of unified memory and a Samsung 9100 PRO 2 TB SSD, running two different large language models. The researchers compared Bevy against the standard method used in popular software like llama.cpp, as well as against the most advanced prediction-based systems available. They found that Bevy consistently outperformed the others. In tests with varying numbers of simultaneous requests, Bevy delivered up to 3.46 times more work per second than the standard method and up to 1.55 times more than the fastest prediction-based system. The improvement was most significant when the computer had to juggle many requests at once. Even when the researchers tested the system with a large cache of books already in memory, where prediction systems struggled with poor accuracy, Bevy maintained its speed. This is because Bevy never wasted time fetching books it didn't need; it only fetched what was explicitly required by the requests moving through the system.

The study also explored whether adding prediction on top of Bevy would help further. The researchers tried to combine the new scheduling method with a prediction system, expecting the two to work together. However, they found that the prediction added no value and sometimes even slowed things down. Because Bevy keeps the storage link so busy fetching the exact books needed for the ready requests, there is no spare time left to fetch predicted books. Furthermore, when the system does try to predict, it often guesses wrong, wasting bandwidth on books that are never used. The researchers concluded that in this specific environment, where a local cache already holds many of the needed items, the act of predicting is not just unnecessary but harmful. The most efficient path is to let the requests reveal their needs as they go and to schedule the work so that the system is always doing something useful, whether it is computing or fetching.

This work suggests a fundamental shift in how local artificial intelligence can be managed. It shows that efficiency does not always come from trying to be smarter about the future, but from being more responsive to the present. By letting tasks proceed at their own pace and grouping them only when it makes sense, the system eliminates the idle time that has long plagued local AI. The findings are based on concrete measurements of throughput and timing across multiple hardware setups, providing a clear picture of how this new scheduling method performs in real-world conditions. The researchers did not claim that this solves every problem of local inference, but they demonstrated that for the specific challenge of running large, memory-hungry models on personal devices, a simple change in the order of operations can yield dramatic improvements. The system, named Bevy, stands as a practical example of how rethinking the flow of work can unlock the full potential of existing hardware, making powerful intelligence faster and more responsive right on the user's desk.