🤖 machine learning

ML Inference Scheduling with Predictable Latency

This paper identifies critical limitations in existing ML inference scheduling approaches, specifically their coarse-grained interference prediction and reliance on static models, which lead to inaccurate latency forecasts and compromised SLOs under dynamic workloads.

Original authors: Haidong Zhao, Nikolaos Georgantas

Published 2026-06-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Haidong Zhao, Nikolaos Georgantas

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The Busy Restaurant Kitchen

Imagine a high-end restaurant kitchen (the GPU) that is incredibly fast but very expensive to run. To save money, the owner wants to cook as many meals as possible at once (improving utilization).

In this kitchen, orders come in for different dishes (like Machine Learning models). To be efficient, the chef groups similar orders together into a single "batch" to cook them all at once. For example, instead of cooking one burger, then two fries, then one burger, they cook a tray of five burgers at the same time.

However, the kitchen has a problem: Interference.
If the chef tries to cook a huge tray of burgers and a tray of delicate soufflés at the exact same time on the same stove, they might get in each other's way. The heat might get uneven, or the chef might have to switch tools too often. This slows everything down, causing the food to arrive late.

In the world of AI, "late food" means latency. If a self-driving car or a video call gets a delayed answer because the kitchen was too crowded, that's a failure. The goal of this paper is to figure out how to schedule these cooking batches so the kitchen stays busy without making the food late.

The Problem: Guessing is Dangerous

The paper argues that current methods for predicting how much "crowding" (interference) will happen are flawed in two main ways:

1. The "Static Snapshot" Problem (Coarse-Grained)

The Analogy: Imagine you are a traffic controller. You look at a map and see a red truck and a blue car currently on the highway. You predict they will drive side-by-side for the whole trip.
The Reality: In the kitchen, the red truck (Batch 1) might leave the highway after 10 seconds, and a giant semi-truck (Batch 3) might jump in and stay for 5 minutes.
The Paper's Claim: Current AI schedulers are like that traffic controller. They look at who is in the kitchen right now and assume they will stay that way. They ignore the fact that batches come and go at different times.

Result: The prediction is wrong. The system thinks the kitchen will be calm, but suddenly a new, heavy batch arrives, causing a traffic jam and late deliveries.

2. The "Old Map" Problem (Non-Adaptive/Static Models)

The Analogy: Imagine you trained a robot chef to predict cooking times using a menu from last year. Last year, everyone ordered burgers. This year, everyone is ordering complex seafood.
The Reality: The robot chef still thinks, "Oh, it's just burgers," and predicts the cooking time will be fast. But because the menu changed, the robot is wrong.
The Paper's Claim: Current AI models are "static." They are trained once on old data and then never updated. If the types of requests change (e.g., new AI models are added, or the number of orders changes), the old model becomes inaccurate.

Result: The system keeps making bad guesses because it's using an outdated map for a changing world.

What the Authors Did (The Experiment)

The authors set up a test kitchen (using an NVIDIA GPU) to see how bad these two problems really are.

Testing the "Snapshot" issue: They ran simulations where batches arrived and left at different times. They found that if you ignore these changes, your prediction of how long a task will take can be off by a huge margin (sometimes more than 60% error). It's like guessing a 10-minute drive will take 16 minutes because you didn't know a traffic light was about to turn red.
Testing the "Old Map" issue: They trained a model on one set of dishes, then tried to predict times for a completely different set of dishes. The old model failed miserably. However, when they let the model "learn on the fly" (Online Learning)—updating its predictions as it saw new data—it got much better at handling the changes.

The Conclusion

The paper concludes that to run an efficient AI kitchen without late deliveries, we need to stop using "dumb" schedulers that:

Ignore the fact that the crowd in the kitchen changes every second.
Refuse to learn from new types of orders.

Instead, we need a smart scheduler that:

Watches the kitchen dynamically (tracking who arrives and leaves).
Learns and updates its predictions in real-time as the workload changes.

By doing this, we can keep the GPU (the kitchen) busy and efficient while still guaranteeing that every order (every AI request) gets its answer on time.

Technical Summary: ML Inference Scheduling with Predictable Latency

Problem Statement
Machine learning (ML) inference serving systems aim to maximize GPU utilization while adhering to strict Service Level Objectives (SLOs) or deadlines. To improve utilization, systems often employ dynamic batching (consolidating requests) and concurrent batch execution (running multiple batches simultaneously on a single GPU). However, concurrent execution introduces interference as batches contend for limited GPU resources (e.g., L2 cache, DRAM, Tensor Cores). This interference creates unpredictable latency deviations, making it difficult to guarantee SLOs.

Existing interference prediction approaches suffer from two primary limitations:

Coarse-Grained Prediction: They often ignore runtime co-location dynamics. A batch may co-locate with different concurrent batches for varying durations during its execution. Static models that assume a single, constant interference source fail to capture these temporal shifts, leading to inaccurate predictions.
Non-Adaptive Models: Existing approaches typically rely on static models trained offline. These models degrade in accuracy when workload characteristics change (e.g., new models are deployed, or request arrival rates shift), necessitating costly retraining and redeployment.

Methodology
The authors evaluate these limitations through a series of simulations and experiments conducted on an NVIDIA L4 GPU using the TensorRT inference engine.

Experimental Setup: The study utilizes a diverse set of DNN models (CNNs like ResNet-50, VGG-19, ConvNeXt-B; and Transformers like RoBERTa-B, ViT-B-16, YOLO-v8n). Workloads follow a Poisson arrival process, ranging from light loads to stress testing.
Baseline Approach: The evaluation adopts the interference prediction method from gpulets [7], which uses a linear regression model ( $\hat{y} = w^T x + b$ ). The input features ( $x$ ) consist of average resource throughput metrics (L2 cache, DRAM, and SM throughput) of the interfered batch and its co-located batch, profiled offline using NVIDIA Nsight Compute.
Evaluation of Coarse-Grained Limitations: The authors simulate scenarios where co-location dynamics change (e.g., Batch 2 co-locates briefly with Batch 1, then for a long duration with Batch 3). They compare:
- Static: Ignoring dynamics entirely (using the initial runtime state).
- Partial Dynamics: Tracking arrivals/departures and smoothing throughput using an Exponentially Weighted Moving Average (EWMA) with varying smoothing factors ( $\alpha$ ).
Evaluation of Non-Adaptive Limitations: The authors compare three learning strategies across different test sets (varying loads, new models, and heterogeneous workloads):
- Offline Learning: A static model trained once on a specific dataset.
- Online Learning (SGD): Updates model weights at runtime using Stochastic Gradient Descent.
- Online Learning (RLS): Updates model weights using Recursive Least Squares.

Key Results

Impact of Co-Location Dynamics: Ignoring runtime co-location dynamics leads to significant prediction errors. Even when partially accounting for dynamics via EWMA, the prediction model still exhibits noticeable deviations. In the evaluation, while over 50% of predictions fell within a 20% relative error, a significant portion exceeded 60% error. The authors attribute this to the use of inaccurate, static co-located resource throughput values as input features, which fail to reflect the actual resource pressure experienced by a batch over time.
Impact of Workload Changes: Static models suffer from performance degradation under changing workloads.
- When tested on newly deployed models (Test Set 2) not seen during training, the offline learning method showed a massive increase in Mean Squared Error (MSE) from 0.0355 to 0.5480.
- Online learning methods (both SGD and RLS) significantly mitigated this degradation, maintaining much lower MSEs (0.4311 for SGD and 0.3467 for RLS).
- RLS consistently outperformed plain SGD, likely due to its ability to exploit curvature information for faster convergence, though at a higher computational cost.

Key Contributions

Evaluation of Prediction Limitations: The paper provides empirical evidence that coarse-grained interference prediction methods, which ignore temporal co-location dynamics, lead to substantial accuracy deviations.
Assessment of Model Adaptability: The study demonstrates that static prediction models degrade considerably under changing workload characteristics (concept drift), whereas online learning approaches can sustain predictive accuracy.
Baseline for Future Work: By rigorously testing the gpulets approach, the authors establish a baseline showing the specific failure modes of current state-of-the-art interference prediction in dynamic ML serving environments.

Significance and Claims
The authors modestly claim that their work highlights critical gaps in current interference prediction strategies. They argue that for ML inference scheduling to achieve high GPU utilization while meeting SLOs, prediction mechanisms must be redesigned to:

Account for runtime co-location dynamics rather than relying on static snapshots.
Incorporate online learning capabilities to adapt to evolving workload characteristics without requiring full retraining.

The paper positions itself as a foundational evaluation ("Work in Progress") that identifies the need for more sophisticated, adaptive prediction models and better resource metrics to enable truly predictable latency in ML inference systems.