DriftSched: Adaptive QoS-Aware Scheduling under Runtime Token Drift for Multi-Tenant GPU Inference

Technical Summary: DriftSched

Problem Statement
The rapid adoption of Large Language Models (LLMs) has intensified the demand for efficient multi-tenant GPU inference serving. In shared infrastructure, heterogeneous requests compete for limited resources while requiring predictable latency and Quality-of-Service (QoS) guarantees. While modern inference runtimes (e.g., vLLM, TensorRT-LLM) improve throughput via continuous batching and optimized memory management, efficient scheduling under sustained contention remains a fundamental challenge. A central difficulty lies in accurately estimating workload cost at admission time. Inaccurate estimates—often derived from static token budgets or lightweight heuristics—can lead to workload misclassification, queue imbalance, fairness degradation, and increased tail latency. This paper investigates the sensitivity of QoS scheduling policies to workload-characterization fidelity and the impact of runtime calibration on scheduling effectiveness.

Methodology and System Architecture
The authors propose DriftSched, a workload-aware QoS scheduling framework for multi-tenant LLM inference. The system architecture comprises:

1. Workload Characterization: Two admission-time strategies are compared:
   • A lightweight whitespace-based proxy (split()) that estimates workload size using word counts.
   • Tokenizer-aware accounting using the model’s native tokenizer for precise token-space estimation.
2. Queue Management: Requests are classified into semantic categories (short QA, summary, technical, report) and mapped to runtime scheduling classes (short, medium, long) based on estimated token budgets. Requests are grouped into tenant-specific queues: Premium, Standard, and Batch.
3. Scheduling Policies: Five policies are evaluated: First-In-First-Out (FIFO), Priority Scheduling, Weighted Scheduling, Shortest-Job-First (SJF), and Aging Priority Scheduling.
4. Adaptive Calibration: An optional online Exponential Moving Average (EMA) mechanism incorporates runtime execution feedback to calibrate workload estimates. This mechanism updates workload-specific bias factors to compensate for systematic estimation errors, particularly when using the lightweight split() proxy.

Experiments were conducted on an NVIDIA L4 GPU using the vLLM inference runtime and the Qwen1.5-1.8B-Chat model. Each experimental run consisted of 3,000 requests, divided into a 1,000-request calibration phase and a 2,000-request stress evaluation phase.

Key Contributions

• The introduction of DriftSched, a framework combining admission-time workload characterization, tenant-aware queue management, and adaptive runtime calibration.
• A comparative study contrasting whitespace-based workload estimation with tokenizer-aware admission-time accounting.
• An empirical evaluation of five scheduling policies (FIFO, Priority, Weighted, SJF, Aging Priority) under heterogeneous multi-tenant GPU workloads.
• An online EMA-based calibration mechanism designed to compensate for admission-time workload-estimation errors.
• A reproducible benchmarking and telemetry framework for studying QoS-aware GPU inference scheduling.

Results

• Workload Characterization Fidelity: Tokenizer-aware accounting provides significantly more accurate admission-time estimates than the whitespace-based proxy. The split() method introduces systematic estimation errors, causing calibration factors to diverge from unity. However, the EMA-based calibration mechanism successfully compensates for these errors, allowing the lightweight estimator to approach tokenizer-aware behavior after convergence. In contrast, tokenizer-aware accounting produces calibration factors that remain close to unity, indicating that most estimation drift originates from admission-time characterization errors rather than intrinsic model variability.
• Scheduling Policy Impact: Scheduling policy selection has a greater impact on overall performance than runtime calibration alone.
  • SJF achieves the strongest latency performance, reducing median end-to-end latency by approximately 42% and P99 latency by approximately 16% relative to FIFO. It prioritizes shorter estimated workloads, significantly reducing queue residence time for latency-sensitive requests.
  • Priority Scheduling provides the strongest tenant-level QoS differentiation, ensuring Premium tenants receive preferential access at the expense of Batch tenants.
  • Weighted and Aging Priority schedulers offer balanced allocation strategies, mitigating starvation while maintaining some service differentiation.
• GPU Utilization: GPU memory consumption and utilization remained relatively constant across all scheduling policies, indicating that performance differences stem from queue management and workload ordering rather than changes in GPU execution efficiency.

Significance and Claims
The paper claims that accurate admission-time workload characterization is a key enabler of effective QoS-aware scheduling for multi-tenant GPU inference systems. It demonstrates that while adaptive runtime calibration can mitigate errors from coarse estimators, tokenizer-aware accounting largely eliminates the need for such correction. Furthermore, the findings suggest that scheduler design exerts a stronger influence on tenant-level QoS outcomes and tail latency than adaptive bias correction alone. The work provides practical guidance for enterprise-scale AI inference platforms, highlighting the tradeoff between global latency optimization (best served by SJF) and tenant-level service guarantees (best served by Priority-based schedulers).