FLINT: Efficiently Leveraging High Bandwidth Flash for Capacity-Scalable LLM Inference Acceleration

The world of artificial intelligence is currently racing toward a physical limit. The most advanced language models, the systems that write code, draft stories, and solve complex problems, have grown so massive that they no longer fit inside the standard memory chips attached to the computer processors that run them. Imagine trying to carry a library of a billion books in a backpack that can only hold a few dozen; this is the current reality for the hardware powering these models. To make these systems work, engineers have been forced to split the models across many separate computers, a solution that is expensive, slow, and energy-hungry because the computers must constantly talk to each other to share the load. A new technology called high-bandwidth flash offers a way to store these massive libraries right next to the processor, providing terabytes of space in a tiny package. However, this new storage is much slower to read than the processor's own memory, creating a bottleneck where the computer spends most of its time waiting for data to arrive.

Researchers at Huawei and ETH Zurich have designed a system called FLINT to solve this waiting problem. Their work, detailed in a recent study, proposes a way to make this high-capacity flash storage run fast enough to keep up with the processor's demands without needing the complex, pre-planned schedules that previous designs required. Instead of guessing which data the processor will need next, FLINT watches the processor in real time and organizes the data requests as they happen. By grouping these requests together and using the storage chip's internal buffers more efficiently, the system can stream data at speeds that nearly match the processor's own memory. The team simulated this system using six different large language models, including some of the largest and most complex models currently in existence. Their results show that this approach allows a single computer package to handle tasks that previously required dozens of machines, while using significantly less energy and delivering results much faster.

The core of the problem lies in how these models are built. To run an inference, which is the act of the model generating a response, the computer must constantly access the model's weights, which are essentially the billions of numbers that define its knowledge. In the past, these weights were small enough to fit entirely in the high-speed memory attached directly to the processor. Now, with models containing hundreds of billions or even trillions of parameters, the weights are too large for that fast memory. The alternative has been to use solid-state drives, which are large and cheap but far too slow, or to use multiple processors, which introduces delays as they synchronize their work. High-bandwidth flash was proposed as a middle ground: a storage technology that fits in the same small package as the processor and offers massive capacity, but with a read speed that is still too slow for the processor to use directly without stalling.

Previous attempts to use this flash storage relied on a method similar to a librarian guessing which books a reader will need next. The system would try to predict the next layer of the model and start fetching those weights before the processor asked for them. The researchers found that this guessing game failed miserably. Because the models are so complex and their behavior changes depending on the specific question being asked, the guesses were often wrong. The system would fetch data that the processor didn't need yet, filling up the limited buffer space and forcing it to throw away data that was actually needed. This resulted in the processor spending most of its time waiting for the correct data to be fetched again, wasting the high speed of the flash storage.

FLINT changes this approach by removing the guesswork entirely. Instead of a librarian guessing, the system acts like a highly efficient traffic controller that watches the flow of cars in real time. When the processor asks for a specific piece of data, FLINT does not just fetch that single piece. It looks at the stream of requests coming from the processor and groups them together. If the processor asks for several pieces of data that happen to be stored in the same physical area of the flash chip, FLINT bundles them into a single, large request. This allows the flash chip to read a massive amount of data in one go, utilizing its full speed. The system then holds this data in the chip's internal buffers, ready to be handed to the processor exactly when it is needed. This dynamic grouping means the system never wastes time fetching data that isn't needed, and it keeps the data pipeline full and flowing.

Another major hurdle for flash storage is that it requires maintenance. Over time, the act of reading data can cause the electrical charges inside the chip to degrade, leading to errors. To fix this, the chip must periodically pause and rewrite the data to fresh locations. In previous designs, this maintenance had to happen while the processor was trying to read data, causing the system to freeze for long periods. The researchers solved this by creating a "phantom plane" system. They added a small amount of extra space to the chip that acts as a spare lane. When a section of the chip needs maintenance, the system quietly copies the data to this spare lane in the background, without ever stopping the main flow of data to the processor. This ensures that the maintenance work never interrupts the processor's work, keeping the system running smoothly even as the chip ages.

The team also simplified the way the system translates the processor's requests into physical locations on the chip. Standard storage systems are built to handle constant writing and rewriting, which requires complex maps and heavy machinery to manage. Since the weights of a language model are written once and never changed, FLINT uses a much simpler, lightweight map that is optimized only for reading. This reduction in complexity means the system can translate requests almost instantly, removing another layer of delay.

When the researchers tested these ideas in a detailed simulation, the results were striking. They compared their new system against three other setups: a standard system using slow solid-state drives, a system using only the fast memory across many processors, and a previous attempt to use flash storage. The FLINT system was able to decode text, which is the process of generating the model's output, up to 1,205 times faster than the system using solid-state drives. Compared to the system that relied on many processors working together, FLINT was more than twice as fast on average. Perhaps most importantly, it achieved this performance using far fewer computers. For a small batch of requests, a single computer equipped with FLINT could do the work that previously required four to eight computers, and it did so while consuming significantly less energy.

The study also looked at how long the system would last. Because the flash storage is constantly being read, the researchers were concerned about how quickly the chips would wear out. They found that their maintenance system, which rewrites data before errors occur, extends the life of the storage to many years, even under heavy use. The system was tested with six different models, ranging from dense models to complex ones that switch between different experts to solve problems, and it performed well across the board. The researchers noted that while the system adds a small amount of physical space to the chip, the cost is negligible compared to the massive gains in speed and capacity.

This work demonstrates that the limitations of artificial intelligence are not just about how smart the models are, but also about how we move data around them. By rethinking how storage interacts with the processor, the researchers have shown that it is possible to build systems that are both massive in capacity and fast in speed. The FLINT system turns a technology that was once considered too slow for real-time use into a practical solution for the next generation of artificial intelligence, allowing powerful models to run on single machines rather than requiring entire data centers. The findings suggest that with the right engineering, the physical constraints of memory can be overcome, opening the door for more capable and efficient AI systems in the future.