Beyond Binary Priorities: Multi-Tier SLA Scheduling for Large Language Model Serving

In the world of modern artificial intelligence, large language models act as powerful engines that generate text, code, and ideas. However, running these engines is not a simple task of turning a switch on and off. When a user sends a request, the system must first process the input and then generate a response one word at a time. This process is unpredictable; a short question might take a fraction of a second, while a complex analysis could stretch for minutes, consuming vast amounts of computer memory along the way. Because these requests arrive in sudden bursts and vary wildly in length, the infrastructure supporting them often struggles to keep up. If the system treats every request exactly the same, a massive, slow task can clog the line, causing urgent, simple questions to wait far too long. This is the central challenge of serving artificial intelligence: how to manage a chaotic stream of work so that critical tasks get immediate attention without wasting the expensive computing power needed to run them.

Researchers at UC Berkeley have tackled this problem by reimagining how these AI systems prioritize their work. They built upon a previous system called Llumnix, which was designed to move tasks between different computer servers to balance the load and prevent bottlenecks. The original version of this system was effective but limited, offering only two levels of importance: high priority for urgent matters and normal priority for everything else. This binary choice was too blunt for real-world businesses, which often need to distinguish between five or more tiers of service, such as platinum, gold, silver, and free users, each with different speed guarantees. The researchers asked a fundamental question: what is the right number of priority levels to create a system that is both fair to all users and efficient for the machine?

To find the answer, the team created a sophisticated simulation of a large-scale AI data center. They did not run this on physical hardware, which would have been prohibitively expensive and slow to test, but instead used a high-fidelity digital model that mimics the behavior of real computer chips. Inside this virtual environment, they expanded the priority system to support anywhere from one to ten distinct levels of importance. They also introduced a new way of managing memory, where each priority level is granted a specific amount of "breathing room" to ensure it can finish its work without being crowded out. This breathing room is not shared equally; instead, it shrinks exponentially as the priority gets lower, ensuring that the most critical tasks always have the space they need while less urgent tasks fill in the gaps.

The team tested this new system against several existing methods, running thousands of simulated requests under different conditions. They varied the mix of users, creating scenarios where high-priority requests were rare, common, or evenly distributed, and they adjusted the total volume of work to see how the system behaved when it was lightly loaded versus when it was nearly full. The results revealed a clear sweet spot. While the system could technically handle up to ten priority levels without collapsing, adding more than four tiers did not improve performance. In fact, beyond four levels, the system began to lose efficiency. The extra complexity of managing so many distinct categories started to consume more resources than it saved, and the benefits of separating the tasks began to fade.

The optimal configuration turned out to be four priority tiers. In their simulations, this setup allowed the system to achieve significant speed improvements for the overall system performance. The end-to-end P99 latency for the system improved by up to 3.13 times compared to standard methods, and the overall cost-per-latency dropped by as much as 68 percent. The system managed to keep the most important users happy by giving them a dedicated lane, while still allowing the less urgent background work to proceed efficiently in the remaining space. This balance prevented the "convoy effect," where a single slow task would hold up an entire line of faster ones, by dynamically moving tasks between servers to keep everyone moving.

The researchers also discovered that the benefits of this fine-grained prioritization depend heavily on how busy the system is. When the data center is moderately loaded, the ability to separate tasks into different lanes works beautifully, allowing high-priority requests to zip through while lower ones wait. However, when the system becomes completely saturated, with every server running at maximum capacity, the advantage of having many priority levels diminishes. In those extreme conditions, the system is so full that there is no extra space to move tasks around, and the distinction between tiers becomes less effective. This suggests that while a four-tier model is the ideal design for most situations, it works best when paired with a system that can automatically add more servers before the load becomes too heavy to manage.

Ultimately, this work provides a practical blueprint for the future of AI infrastructure. It demonstrates that a simple, four-level hierarchy is sufficient to capture the full range of user needs, from critical real-time interactions to background batch processing. By moving away from a rigid two-level system and avoiding the trap of over-complicating the design with too many tiers, engineers can build AI services that are both faster and more cost-effective. The study confirms that the key to handling the unpredictable nature of artificial intelligence lies not in making the system more complex, but in finding the precise level of organization that allows it to breathe.