Efficient INT8 Inference of Small NLP Models on Server CPUs with PyTorch Native Stack
This paper presents a native PyTorch implementation of SmoothQuant for INT8 inference of small NLP models on server CPUs, achieving up to 5.8x throughput speedup with negligible accuracy loss through graph-level fusion and optimized kernel selection, with the solution now upstreamed to PyTorch and TorchAO.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the world of artificial intelligence, a quiet revolution is happening alongside the loud headlines about massive, all-knowing chatbots. While the public eye is fixed on enormous models that can write poetry or solve complex logic puzzles, the engines that power the internet's daily traffic—search results, product recommendations, and spam filters—rely on smaller, specialized programs. These programs, often based on a family of designs known as BERT, are the workhorses of the digital economy. They run on standard computer servers, the kind found in data centers around the globe, rather than on specialized, expensive graphics cards. For these systems, speed and cost matter more than raw size. The challenge for engineers has long been how to make these models run faster without sacrificing the accuracy that keeps them useful. One powerful way to do this is by simplifying the math the computer performs, trading a tiny bit of precision for a massive gain in speed, a technique known as quantization.
A team of researchers at Intel Corporation has now built a bridge that allows these smaller language models to run with this simplified math directly inside the most popular software tools used by developers today. Their work focuses on a specific method called SmoothQuant, which prepares a model to handle the simplified math without losing its ability to understand language. By integrating this method into the standard PyTorch software stack, they have created a path that requires no special third-party tools or complex rewrites. The result is a system that can take a standard language model and make it run significantly faster on modern server processors, all while keeping the answers it gives just as correct as before.
The researchers started with a common problem: even though modern computer processors have built-in instructions to handle simplified math quickly, the software tools used to build these models often did not know how to use them efficiently. When a model was prepared for speed, the software would often get stuck on the steps needed to convert data or manage the simplified calculations, wasting the very speed the hardware promised. To fix this, the team combined three distinct pieces of technology into a single, seamless workflow. First, they used a tool called TorchAO to prepare the model, smoothing out the data so it could be safely simplified. Next, they used a compiler called TorchInductor to look at the entire flow of the model's calculations and merge them together, removing unnecessary steps that usually slow things down. Finally, they taught the system to choose the absolute fastest way to perform the core math operations, depending on whether the computer processor was a newer model with advanced capabilities or an older, reliable model still widely in use.
To test their creation, the team ran experiments on two different generations of Intel server processors. They tested the system on three different types of language models, ranging from large, complex versions to smaller, more compact ones. The results were striking. On the newer processors, the system ran up to 5.8 times faster than the standard, unoptimized version. On the older processors, it still managed to run nearly three times faster. Perhaps most importantly, the speed did not come at the cost of quality. When the researchers tested the models on real-world tasks, such as answering questions from a text or judging the sentiment of a sentence, the simplified models got the answers right just as often as the original, slower versions. In many cases, the difference in accuracy was so small it could not even be measured.
The team also looked closely at why the speed gains happened, using a method that maps out the limits of how fast a computer can move data versus how fast it can calculate. They found that for the newer processors, the speed was limited by how fast data could be moved around, while for the older processors, it was limited by how fast the math could be done. Their system managed to navigate both of these limits effectively. They discovered that by preparing the data in advance and removing the need for the computer to stop and reorganize information while it was working, they could unlock the full potential of the hardware. They even found a clever workaround for older processors that lacked a specific instruction for simplified math, allowing them to use a different, available instruction to achieve similar results.
This work matters because it removes a significant barrier for companies that want to deploy these models efficiently. Previously, getting this level of performance often meant using specialized tools that were difficult to maintain or required leaving the standard software environment. Now, developers can simply use the tools they already know to get the same high performance. The researchers have shared their code with the broader community, meaning that anyone using these standard tools can now access this speed boost immediately. The study confirms that it is possible to make the workhorses of the internet faster and more efficient without changing the fundamental nature of the models or the software they run on. By carefully optimizing how the computer handles the math, the team has shown that the future of efficient artificial intelligence on standard servers is not just a possibility, but a reality that is ready to be used today.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.