Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooning

Prism is a memory-centric LLM co-serving framework that utilizes a novel memory ballooning technique called kvcached to dynamically reclaim and reallocate GPU memory across multiple models, thereby unifying spatial and temporal sharing to improve cost-efficiency and SLO adherence in production environments.

The Gist

Imagine you run a massive hotel with thousands of rooms (GPUs) and thousands of different guests (AI models). Some guests are celebrities who want a room 24/7, while others are tourists who only show up for a 10-minute check-in once a day.

The problem is that your hotel is expensive to run. If you give every tourist their own private room just in case they show up, you end up with 90% of your hotel empty and wasted. But if you try to squeeze everyone into one room, you get chaos, and the celebrities get angry because they have to wait.

Prism is a new, smart hotel manager that solves this by using a trick called "Memory Ballooning."

Here is how it works, broken down into simple concepts:

1. The Problem: The "Static Room" Trap

In the old way of running AI, if a model (a guest) was assigned a room, that room was theirs forever, even if they were sleeping (idle).

• Space Sharing (The Old Way): You try to put multiple guests in one room. It works great if they are all awake and chatting. But if one guest leaves for a week, their half of the room sits empty, and the other guest can't use it.
• Time Sharing (The Other Old Way): You kick one guest out to let another in. This works if guests only come at different times. But if two guests arrive at the exact same second, you have to constantly kick them in and out of the room. This "kicking" is slow and makes everyone wait (lag), causing them to miss their deadlines.

Real-world AI traffic is messy. Sometimes a group of models all get busy at once, and sometimes they all go quiet. No single old strategy could handle this switching.

2. The Solution: The "Ballooning" Trick

Prism introduces a new manager called kvcached (the balloon driver). Think of the GPU memory not as a set of fixed rooms, but as inflatable balloons.

• The Elastic Balloon: When a model is busy and needs more space to think, the manager inflates its balloon, stealing empty air from other models that are currently sleeping.
• Deflating for Others: When a model goes to sleep, its balloon shrinks, releasing that space so a new, waking-up model can instantly inflate its own balloon.
• No Moving Furniture: The best part? The models don't even know this is happening. They just see a room that magically expands and contracts. The manager handles the heavy lifting behind the scenes.

3. The Two-Step Strategy

Prism uses two smart rules to decide who gets the air:

• Rule 1: The Global Scheduler (The Hotel Manager): This looks at the whole hotel. It asks, "Which group of guests is currently active?" It then places those active guests on the same floor (GPU) so they can share space easily. If a guest is sleeping, it moves them to a storage closet (CPU) to free up space. It constantly rearranges the hotel to make sure no floor is overcrowded while another is empty.
• Rule 2: The Local Scheduler (The Concierge): This looks at the specific requests coming in right now. If two guests are fighting for the last bit of space, the concierge checks who has the most urgent deadline. It lets the urgent guest in first and tells the less urgent one to wait a moment. This ensures the most important tasks get done on time.

4. The Results

The paper tested Prism on real-world data from major AI providers and found:

• Faster Service: It met its speed promises (SLOs) up to 3.3 times better than previous methods.
• Cheaper Costs: To get the same level of performance, Prism needed half the number of GPUs (or could handle twice as many requests with the same hardware).
• Real-World Proof: It has already been deployed in production environments with over 10,000 GPUs, helping companies generate significantly more revenue per GPU by turning wasted "idle" time into billable work.

Summary

Prism is like a smart, elastic hotel manager. Instead of locking guests into fixed rooms or kicking them out constantly, it uses inflatable balloons to dynamically share space. It expands space for busy models and shrinks it for sleeping ones, ensuring the hotel is always full, efficient, and fast, without anyone waiting in line.

Technical Summary

Technical Summary: Prism

Problem Statement

Inference providers hosting thousands of Large Language Models (LLMs) face a critical efficiency challenge: maintaining availability for low-volume, essential models while minimizing costs as token prices decline. Production trace analysis reveals that real-world workloads exhibit a dynamic bursty-group pattern, where specific subsets of models become active together and shift over time, driven by compound AI systems and agentic pipelines.

Existing serving strategies fail to adapt to this variability:

• Space Sharing (Colocation): Co-locating multiple models on a GPU to utilize unused memory works for continuous low-traffic models but creates severe memory fragmentation during long idle periods. It prevents active models from scaling up to meet Service Level Objective (SLO) requirements because memory is statically locked by idle models.
• Time Sharing (Swapping): Swapping models in and out of GPU memory handles sporadic traffic well but suffers from high latency overhead (seconds) during rapid workload fluctuations. This causes "thrashing" when multiple models experience interleaved requests, leading to SLO violations.

The core challenge is the lack of a principled mechanism to unify spatial and temporal sharing. Current systems force a trade-off between SLO adherence and resource efficiency, unable to fluidly transition between strategies as workload patterns shift from volatile bursts to interleaved activity.

Methodology

Prism is a memory-centric GPU sharing framework designed to unify spatial and temporal sharing through elastic memory allocation. Its architecture consists of three primary components:

1. GPU Memory Ballooning (The kvcached Driver)

Prism introduces a balloon driver, kvcached, which sits between inference engines (e.g., SGLang) and GPU physical memory. This driver implements memory ballooning to reclaim and reallocate memory across models without disrupting execution.

• Virtual/Physical Decoupling: Engines reserve a large virtual address space, but physical memory is allocated and mapped on demand. This allows the system to shrink or expand a model's physical footprint (weights and KV cache) at runtime.
• Unified Management: kvcached treats model weights and KV cache as a unified pool of physical pages. It supports diverse model architectures by mapping token blocks to virtual pages and segregating them physically to avoid shape conflicts.
• Optimizations: To minimize overhead, it uses 2MB memory pages, prioritizes partially filled pages, and employs a pre-allocation thread for asynchronous page preparation. It exposes an elastic tensor (eTensor) abstraction compatible with existing attention kernels and CUDA graphs, requiring zero code changes to serving engines.
• Fast Loading: To mitigate cold-start latency, Prism decouples engine lifecycles from model lifecycles using a reusable engine pool and employs parallel weight loading across GPUs via NVLink to saturate interconnect bandwidth.

2. Memory-Centric Control Plane

Prism employs a hierarchical scheduler to manage the elasticity provided by kvcached:

• Load-Aware Model Placement (Global): A global scheduler places models across GPUs to maximize memory headroom for ballooning. It uses a KV Pressure Ratio (KVPR) heuristic, which quantifies memory contention based on SLO-weighted token rates. The algorithm greedily places high-demand models on GPUs with the lowest existing pressure, ensuring that active models have room to expand their KV caches during bursts.
• Slack-Aware Request Arbitration (Local): A GPU-local scheduler manages concurrent requests from colocated models. Instead of static isolation, it uses a shared request queue and applies the Moore-Hodgson algorithm to minimize deadline misses. It prioritizes requests based on their time slack (the difference between the deadline and required execution time), ensuring that urgent, short-latency requests are served before longer-running ones, thus maximizing TTFT (Time-to-First-Token) SLO attainment.

3. System Integration

Prism is implemented as a library extending SGLang, utilizing CUDA Virtual Memory Management (VMM) APIs. It operates as a separate control process that coordinates model evictions, activations, and migrations based on runtime metrics.

Key Contributions

1. Unified Sharing Mechanism: Prism is the first system to unify spatial and temporal sharing under a single scheme by treating GPU memory as an elastic resource. It allows the system to act as a time-sharing system (evicting idle models) and a space-sharing system (co-serving active bursts) simultaneously.
2. kvcached Balloon Driver: An open-source driver that enables flexible, fine-grained (2MB) memory redistribution across heterogeneous LLMs with millisecond-level overhead and zero kernel modifications.
3. Production-Scale Deployment: The system has been deployed in production environments across 10,000+ GPUs and is actively maintained.
4. Comprehensive Evaluation: The paper provides a rigorous analysis of production traces from major providers (Hyperbolic, Novita AI) and evaluation platforms (Chatbot Arena), covering 58 heterogeneous models over periods ranging from 11 days to 16 months.

Results

Evaluated on a cluster of up to 32 NVIDIA H100 GPUs using real-world traces:

• SLO Attainment: Prism achieves up to 3.3× higher TTFT SLO attainment and 2× higher TPOT SLO attainment compared to state-of-the-art baselines (including MuxServe++, QLM, and ServerlessLLM) given the same number of GPUs.
• Cost Efficiency: To achieve the same level of SLO attainment, Prism reduces the required GPU count by 2× (e.g., achieving 99% TTFT SLO with 16 GPUs vs. 32+ for baselines) or handles 3.5× more requests with the same hardware.
• Production Impact: In shadow deployments at two companies, Prism increased per-GPU throughput by 3.89× (Company A) and revenue per GPU by 2.86× (Company B) by converting idle capacity into billable tokens without violating SLOs.
• Latency: Model activation latency is reduced to under 1.5 seconds even for 70B models, and migration overhead is negligible (tens of milliseconds) when using NVLink.

Significance

The paper argues that the fundamental bottleneck in multi-LLM serving is GPU memory contention, not compute. By shifting the perspective from static partitioning or rigid time-slicing to elastic memory management, Prism resolves the conflict between the need for immediate responsiveness (requiring colocation) and the need for resource efficiency (requiring eviction). The work demonstrates that a memory-centric view can fluidly adapt to the "bursty-group" nature of modern AI workloads, enabling providers to host thousands of models cost-effectively while strictly adhering to latency SLOs. The open-sourcing of kvcached and its adoption in large-scale production environments validate the practical viability of this approach.