Strait: Perceiving Priority and Interference in ML Inference Serving

1. Problem Statement

Machine Learning (ML) inference serving systems face significant challenges when deployed in on-premises scenarios under high GPU utilization, specifically regarding task prioritization and deadline satisfaction.

• Insufficient Prioritization: Existing systems (e.g., TensorFlow Serving, NVIDIA Triton) often lack robust support for prioritizing critical tasks (e.g., safety monitoring, quality control) over less critical ones. While they may support CUDA stream priorities, these hardware hints are often ineffective because concurrent kernels still contend for shared resources, potentially causing high-priority (HP) tasks to suffer severe interference or miss deadlines.
• Unpredictable Interference: Current scheduling often relies on temporal sharing (sequential execution) or static spatial sharing (concurrent execution).
  • Temporal sharing suffers from Head-of-Line (HOL) blocking and underutilization.
  • Spatial sharing improves utilization but introduces interference during concurrent kernel execution and data transfers. This interference is non-linear and difficult to estimate, leading to unpredictable latency and deadline violations.
• Lack of Preemption: Commodity GPUs generally do not support true task-level preemption. Once a batch is dispatched, it cannot be interrupted. Therefore, scheduling decisions must be made proactively to avoid violating deadlines, requiring accurate latency estimation under interference.

2. Methodology: The Strait System

Strait is an inference serving system designed to enhance deadline satisfaction for dual-priority traffic (High-Priority and Low-Priority) by explicitly modeling and managing interference.

A. System Architecture

Strait operates locally on a node with multiple GPUs. It consists of:

1. Priority-Aware Scheduler: Manages task queues for different models, assigning them HP or LP status.
2. Interference Prediction Engine: A global model that estimates latency delays caused by data transfer and kernel execution interference.
3. Adaptive Learning Loop: Continuously updates the prediction model based on runtime feedback to handle workload drift and hardware variations.

B. Interference Modeling

Strait decomposes inference latency ($T_{inf}$) into four components:
$$T_{inf} = T_{isol} + T_{data} + T_{kernel} + T_{queue}$$

1. Data Transfer Interference ($T_{data}$):

   • Modeled based on the observation that PCIe data transfers with pinned memory follow a FIFO process.
   • Delays are calculated based on the completion time of the previous batch on the PCIe link.
   • Downstream transfer interference is often negligible due to small output sizes.

2. Kernel Execution Interference ($T_{kernel}$):

   • Prediction Model: Uses an adaptive exponential growth model to estimate slowdown. The intuition is that interference increases slowly at low resource pressure but accelerates super-linearly as pressure mounts (exceeding the GPU's ability to hide latency).
   • Inputs: The model uses time-weighted averages of resource throughput metrics (e.g., L1/L2 cache, DRAM, Tensor Cores, CUDA Cores) from co-located batches.
   • Priority Coefficient: Includes a learnable parameter ($coeff_p$) to reflect that HP tasks may experience less interference than LP tasks under identical pressure due to CUDA stream prioritization.
   • Adaptation: The model uses Adam optimization and Huber loss to update parameters online. It recalibrates based on the discrepancy between predicted and actual kernel execution latency, ensuring robustness against concept drift (e.g., changing model architectures or GPU types).

C. Priority-Aware Scheduling Algorithm

The scheduler employs a binary search to determine the optimal batch size and GPU assignment. It enforces two main constraints:

1. No Violation of Higher/Equal Priority: A new batch is only scheduled if it does not cause ongoing HP or equal-priority tasks to miss their deadlines.
2. Self-Deadline Feasibility: The new batch must be able to meet its own deadline, estimated conservatively (assuming it runs at half the GPU's runtime throughput).
3. LP Throttling: To protect HP tasks, the system dynamically caps the aggregate throughput of LP tasks ($C_{low}$) using an Additive-Increase Multiplicative-Decrease (AIMD) policy. If an HP task misses a deadline, $C_{low}$ is reset to a conservative baseline.

3. Key Contributions

1. Interference-Aware Scheduling: Strait is the first system to integrate explicit interference prediction (both data transfer and kernel execution) directly into the scheduling decision for dual-priority ML traffic.
2. Adaptive Prediction Model: Proposes a novel, self-adapting regression model that learns resource contention patterns dynamically, overcoming the limitations of static models that fail when workloads or hardware change.
3. Priority Management without Preemption: Demonstrates how to achieve effective task prioritization on non-preemptive GPUs by proactively blocking or throttling low-priority tasks to protect high-priority deadlines.
4. Comprehensive Evaluation: Provides a rigorous evaluation against state-of-the-art baselines, including temporal sharing, static/reactive spatial sharing, and software-defined preemption (XSched).

4. Evaluation Results

Experiments were conducted on NVIDIA L4 GPUs using diverse DNN models (ResNet, ViT, YOLO, RoBERTa) under stochastic, uniform, and production traces.

• Deadline Satisfaction:
  • Strait reduces deadline violations for High-Priority tasks by 1.02 to 11.18 percentage points compared to existing spatial sharing and preemption approaches.
  • It incurs an acceptable performance trade-off for Low-Priority tasks (worst-case degradation of only 0.61 percentage points).
• Comparison with Preemption (XSched):
  • Compared to XSched (a kernel-level preemption system), Strait achieves comparable HP deadline satisfaction but offers more equitable performance for LP tasks.
  • XSched suffers from HOL blocking for LP tasks when HP requests dominate, whereas Strait's proactive scheduling avoids this by managing concurrency and batch sizes.
• Adaptability:
  • The adaptive prediction model significantly outperforms static models when transitioning between different model sets (CNN vs. Transformer) or GPU architectures (Ada Lovelace vs. Ampere vs. Hopper), reducing p99 prediction errors by 7.2 to 8 percentage points.
• Overhead:
  • Scheduling overhead is minimal (tens of microseconds), making it suitable for real-time serving.

5. Significance

Strait addresses a critical gap in on-premises ML infrastructure where deterministic latency and task prioritization are paramount (e.g., industrial automation, safety systems). By moving from reactive resource management to proactive interference perception, Strait enables:

• Higher Reliability: Ensuring critical tasks meet strict deadlines even under heavy load.
• Better Resource Utilization: Allowing safe spatial sharing without the unpredictability that usually forces systems to rely on inefficient temporal sharing.
• Scalability: The approach is orthogonal to cluster-level autoscaling and can be integrated into existing serving stacks (like Triton) to enhance their priority handling capabilities without requiring specialized hardware preemption support.

In summary, Strait provides a robust framework for managing the complex trade-offs between concurrency, interference, and priority in modern GPU-accelerated inference serving.