Technical Summary: Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption

Problem Statement

Modern Large Language Model (LLM) deployments increasingly require serving multiple heterogeneous models (varying in architecture, size, and specialization) on shared, resource-constrained hardware. While existing systems like vLLM optimize throughput for single-model workloads via continuous batching and efficient Key-Value (KV) cache management, they lack mechanisms to handle multi-model scheduling under strict GPU memory constraints.

In such constrained environments, two critical mechanisms become necessary but are poorly understood in a multi-model context:

1. Partial CPU-GPU Offloading: Moving subsets of model layers to CPU memory to fit models into limited GPU VRAM.
2. Preemption: Pausing a running inference job to evict its model and state, allowing a higher-priority job to execute, and then resuming the original job.

Current literature often assumes linear performance degradation with offloading or treats preemption costs as a monolithic, unquantified overhead. There is a lack of empirical data characterizing how these factors interact across different model scales, architectures, and hardware platforms.

Methodology

The authors conducted a systematic empirical study across two distinct hardware platforms sharing a common CPU subsystem (AMD Threadripper PRO 5995WX) but differing in GPU architecture:

• Platform A: NVIDIA RTX 5000 Ada Generation (32 GB VRAM).
• Platform B: NVIDIA RTX A6000 (48 GB VRAM).

The study utilized a diverse set of models (Llama 3 8B, Qwen3 32B, Llama 2 70B for offloading; Qwen2.5-3B, Qwen3-8B, Qwen2.5-14B for preemption) and executed two primary experimental sweeps:

1. Layer-Wise Offloading Sweep:

   • Executed on Ollama (Q4 quantized) and HuggingFace Transformers (FP16).
   • Varied the fraction of transformer layers placed on the GPU from 0% (all CPU) to 100% (all GPU) in 10% increments, with finer granularity near 100%.
   • Measured decode throughput (tokens/second) across various output lengths (50 to 5,000 tokens).

2. Preemption and Migration Benchmark:

   • Simulated a "worst-case" preemption scenario where a long-running Job A (generating 7,000 tokens) is interrupted at nine checkpoints (100 to 5,000 tokens).
   • Job A is fully unloaded (weights and KV cache moved to host memory), a Job B is executed, and Job A is reloaded and resumed.
   • Decomposed the total preemption overhead into constituent steps: KV cache transfer (GPU→CPU), model unloading, model reloading, and KV cache restoration (CPU→GPU).
   • Tested various model pairings to isolate the impact of the preempted model's size versus the preempting model's size.

Key Contributions

The paper provides three primary contributions:

1. Systematic Empirical Evaluation: A comprehensive characterization of state-of-the-art LLMs across heterogeneous hardware, demonstrating that performance trends regarding offloading and scaling are consistent across models and systems.
2. Quantification of Overheads: Detailed analysis of the performance implications of partial offloading and preemption, identifying the dominant sources of latency and data movement costs.
3. Scheduler Feature Distillation: A set of key features derived from empirical data that future multi-model schedulers must incorporate to support hybrid CPU-GPU execution and preemption effectively.

Key Results

1. Non-Linear Offloading Degradation

• Model Sensitivity: Decode throughput does not scale linearly with the percentage of layers on the GPU. Smaller models (e.g., Llama 3 8B) exhibit sharp, non-linear sensitivity to reduced GPU residency, suffering significant throughput penalties even with moderate offloading. Larger models (e.g., Llama 2 70B) show a more gradual, approximately linear degradation.
• Hardware Interaction: The relative performance gap between CPU and GPU significantly impacts offloading efficiency. On the RTX 5000 (lower peak throughput), the relative penalty of offloading is smaller compared to the RTX A6000 (higher compute capability), where offloading incurs a larger relative degradation.
• Sequence Length: Longer prompts amplify the negative impact of offloading due to increased KV cache memory pressure and access costs.

2. Preemption Overhead Composition

• Dominance of Model Reload: The cost of preemption is overwhelmingly dominated by reloading model weights from disk to GPU memory. This component accounts for >98.5% of the total preemption overhead on both hardware platforms.
• KV Cache Negligibility: KV cache transfer (both GPU→CPU and CPU→GPU) contributes less than 1.5% of the total overhead, even for large caches (up to ~950 MB).
• Constant Cost: Preemption overhead is effectively constant regardless of the job's progress (number of tokens generated). The cost depends almost entirely on the weight footprint of the preempted model, not the size of the KV cache or the identity of the preempting model.
  • Example: On RTX 5000, preemption overhead ranges from ~3s (3B model) to ~7.3s (14B model).
• Hardware Variance: The RTX A6000 demonstrated lower absolute preemption costs (faster disk-to-GPU transfer) compared to the RTX 5000, despite having larger VRAM.

3. Data Movement Efficiency

• KV cache transfer times scale linearly with sequence length but remain in the tens of milliseconds (e.g., <90ms for ~950MB transfer on RTX 5000).
• Effective PCIe bandwidth is limited by per-layer tensor copy overheads in the PyTorch runtime, reaching only 9–16 GB/s against a theoretical 31.5 GB/s peak.

Significance and Claims

The paper argues that these empirical findings fundamentally alter the design requirements for next-generation LLM schedulers:

• Rejection of Linear Assumptions: Schedulers cannot rely on proportional or uniform placement strategies. They must account for model-specific, non-linear offloading sensitivity, particularly for smaller models where partial offloading is highly detrimental.
• Fixed Preemption Penalty: Contrary to the assumption that preemption costs vary with job progress, the study establishes that preemption can be modeled as a fixed penalty dependent only on the model's weight footprint and the hardware pair. This simplifies scheduling logic, allowing for predictable cost modeling.
• Hardware-Awareness: Scheduling decisions must be explicitly hardware-aware, considering not just GPU capacity but the specific compute gap between the CPU and GPU, as well as disk-to-GPU transfer speeds.
• KV Cache vs. Weights: The study clarifies that optimizing KV cache migration is secondary to optimizing model weight loading/unloading for preemption efficiency, as the former is currently negligible in the total cost structure.

The authors conclude that future schedulers must integrate these features—non-linear offloading curves, model architecture profiles, sequence length dynamics, and fixed preemption costs—to efficiently manage heterogeneous, multi-model workloads in dynamic, resource-constrained environments.