NeuroPrefetcher: Storage-Aware Sparse LLM Inference via Delta Prefetching
NeuroPrefetcher is a storage-aware LLM inference system that achieves significant speedups on memory-constrained edge devices by exploiting the temporal locality of MLP neuron activity to predictively prefetch only the necessary weight deltas from storage, rather than relying on reactive demand paging.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Large language models are the engines behind modern artificial intelligence, capable of writing, reasoning, and translating with a fluency that once seemed impossible. These systems work by processing text one word at a time, predicting the next word in a sequence based on the vast patterns they learned during training. To do this, they rely on massive digital libraries of numbers, called weights, which are stored in the computer's memory. The larger and more capable the model, the more memory it requires. However, a significant gap has emerged: while these models have grown exponentially in size, the memory available in portable devices like laptops, tablets, and specialized edge computers has not kept pace. This creates a fundamental problem for running advanced intelligence outside of massive data centers. When a model is too big to fit into a device's memory, the system must constantly swap data between the fast memory and the slower, larger storage drive, a process that usually grinds performance to a halt.
Researchers at Kennesaw State University and Samsung have developed a new approach to this problem, creating a system called NeuroPrefetcher that allows these oversized models to run efficiently on devices with very limited memory. Instead of trying to shrink the model or force it to fit, their system accepts that the model is larger than the available memory and treats the storage drive as an active part of the calculation process. The key to their success lies in a simple observation about how these models think. When the model generates one word, it activates a specific set of internal pathways. When it generates the very next word, it uses almost the exact same pathways again. The researchers found that between 82 and 85 percent of the active pathways remain the same from one word to the next. This means that for the vast majority of the work, the necessary data is already sitting in the device's memory, waiting to be used.
The innovation of NeuroPrefetcher is in how it manages the small amount of data that does change. Traditional systems wait until the model needs a piece of data before it goes to the storage drive to fetch it, a reactive process that causes the computer to pause and wait. NeuroPrefetcher works differently by looking ahead. It uses a small, specialized predictor that analyzes the current word and guesses exactly which new pathways will be needed for the next word. Because it knows what is coming, it can fetch only the specific, missing pieces of data from the storage drive while the computer is busy calculating the current word. This turns a chaotic, stop-and-go process into a smooth, continuous flow. The system essentially pre-loads only the tiny fraction of new information required, ignoring the massive bulk of data that is already resident in memory.
To test this idea, the team built a complete system and ran it on a powerful edge device known as the Jetson AGX Orin, which has a unified memory architecture shared between its processor and graphics unit. They tested the system with large language models like Mistral-7B and Llama-3-8B under strict memory limits, simulating conditions where the model was significantly larger than the available memory. In these challenging conditions, the standard method of running these models, which relies on the operating system to manage data swapping, performed poorly, often stalling the device. In contrast, NeuroPrefetcher kept the device moving, achieving a speedup of nearly eight to twelve times faster than the standard method. The system managed to generate words at a rate of over three per second, whereas the standard approach struggled to produce even one word per second.
The researchers also examined how the system behaved as the available memory changed. They found that the system could adapt by shifting more of the model's work to the storage drive when memory was tight, and shifting more work back to the fast memory when more space was available. Even in the tightest memory conditions, the system maintained high performance because it avoided the massive data transfers that usually slow down these operations. Instead of moving entire layers of the model's brain back and forth, it moved only the small, changing slices of data. This approach proved so effective that the system remained the fastest option even when compared against other advanced tools, many of which failed to run at all under these specific constraints.
A critical part of this work was ensuring that the speed gains did not come at the cost of intelligence. The researchers measured the quality of the text produced by the system and found that it remained very close to the quality of the full, uncompressed model. The system preserved the accuracy of the model's predictions, retaining over 90 percent of the original performance even when using the most aggressive memory-saving settings. This confirmed that the strategy of moving only the necessary data did not degrade the model's ability to understand and generate language. The system effectively learned to ignore the data that wasn't needed for the immediate next step, focusing its resources only on what mattered.
The study highlights a shift in how we might think about running artificial intelligence on everyday devices. For years, the solution to running large models on small hardware has been to make the models smaller or to compress them, which can sometimes reduce their capabilities. NeuroPrefetcher suggests a different path: keep the full model intact and manage the movement of its data with extreme precision. By predicting what the model needs next and fetching only that specific change, the system turns the storage drive from a bottleneck into a helpful partner. This approach allows powerful intelligence to operate on devices that were previously too small to hold it, opening the door for advanced AI to run locally without needing a connection to a distant server. The results show that with the right management of data flow, the physical limits of memory can be overcome without sacrificing the power of the model itself.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.