💻 computer science

Load Testing for Machine Learning Model Serving Systems at Scale

This paper introduces \sys, an industrial load testing framework that employs an adaptive, feedback-driven search strategy to systematically estimate GPU capacity for ML serving systems, demonstrating through 14 case studies that it significantly improves resource efficiency and operational reliability by reducing estimation errors and preventing SLO violations.

Original authors: Amr S. Abdelfattah, Nakul Tirumalai, Indu Mohanan, Xiao Li, Pengchao Wang, Dinakar Dhurjati, Eric Sung

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Amr S. Abdelfattah, Nakul Tirumalai, Indu Mohanan, Xiao Li, Pengchao Wang, Dinakar Dhurjati, Eric Sung

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the manager of a massive, high-tech restaurant kitchen. This kitchen doesn't cook food; it processes millions of complex math problems every second using powerful graphics cards (GPUs) to run Artificial Intelligence (AI) models.

The big problem? You don't know exactly how many chefs (GPU resources) you need.

If you hire too few, the kitchen gets overwhelmed, orders get delayed, and customers get angry (this is called violating "Service Level Objectives" or SLOs).
If you hire too many, you are paying for empty chairs and idle chefs, wasting a huge amount of money and energy.

For a long time, figuring out the right number of chefs was a guesswork game. This paper introduces a new system called Vanguard that acts like a super-smart "stress test" manager to find the perfect number of chefs.

Here is how Vanguard works, explained through simple analogies:

1. The Problem with Old Tools

Standard stress-testing tools (like JMeter or k6) are like generic fitness trainers. They are great for testing a human running on a treadmill, but they don't understand the quirks of an AI kitchen.

The "Warmup" Issue: When you turn on a high-performance GPU, it's like a race car engine. It needs a few minutes to heat up, cache its parts, and get ready. If you test it immediately, it looks slow and sluggish. Old tools think the engine is broken; Vanguard knows to wait for the engine to warm up before judging its speed.
The "Batching" Issue: AI systems often group requests together (like a bus picking up passengers) to be efficient. If the bus is half-empty, it's fast. If it's full, it might slow down. This relationship isn't a straight line; it's a curve. Old tools assume a straight line; Vanguard understands the curve.
The "Hardware" Issue: A model might run perfectly on one type of GPU but struggle on another. Vanguard tests the specific hardware you actually use.

2. How Vanguard Works: The "Smart Search"

Instead of just guessing a number and hoping for the best, Vanguard uses a feedback-driven search strategy. Think of it like tuning a radio to find the clearest station.

The Adaptive Search: Vanguard starts with a low number of requests. It slowly turns up the volume (adds more requests).
Dampening (The Shock Absorber): As it gets close to the limit where the system might crash, it slows down its steps. It doesn't slam on the brakes; it gently eases off to avoid overshooting the limit.
Spike Tolerance (Ignoring the Noise): Sometimes, the system has a tiny, momentary hiccup (a "spike"). A dumb system might panic and stop testing. Vanguard ignores these tiny blips, knowing they are just noise, and keeps going until it sees a real, sustained problem.
Convergence (Knowing When to Stop): It keeps testing until it's sure it has found the "sweet spot"—the maximum number of requests the system can handle without breaking its promises to the user.

3. The "Health Check" Engine

Vanguard doesn't just look at one number (like speed). It looks at a dashboard of vital signs, similar to a doctor checking a patient.

It checks if the system is Healthy (plenty of room to breathe).
It checks if it's Warning (getting close to the limit).
It checks if it's Critical (about to crash).
To avoid false alarms (like a heart rate monitor glitching), it uses a "hysteresis" rule: the system must stay in a "Warning" state for a few minutes before the system officially declares it's in trouble. This prevents panic over temporary glitches.

4. What They Found (The Results)

The team tested Vanguard on 14 different AI models (like recommendation engines, image recognizers, and text generators) at Meta. Here is what they learned:

Real Traffic is King: The biggest mistake people make is using fake, made-up data to test. The paper found that using recorded real-world traffic (replaying actual user requests) reduced errors from 30% down to just 2–6%. It's the difference between testing a car on a smooth track versus testing it on the actual bumpy road it will drive on.
Warmup Matters: Ignoring the "warmup" period caused a 22% error in predictions. You simply cannot judge a car's top speed the second you turn the key.
The "Crowded Room" Effect: When multiple models share the same GPU (co-location), they interfere with each other, like people talking over each other in a crowded room. This is a major source of error that is hard to predict.
Accuracy: With all the right settings, Vanguard predicted the capacity with 94% accuracy.
Real-World Impact: By using Vanguard, the company was able to cut down on wasted GPU resources by 15% to 83% for different models and significantly reduced the number of times their services crashed due to being understaffed.

5. The Takeaway

The paper concludes that you can't just use generic tools to test AI. You need a specialized approach that understands:

Warmup: Let the system get hot before testing.
Real Data: Test with real traffic, not fake data.
Smart Patience: Don't panic at small glitches; look for sustained trends.

By following these rules, companies can save massive amounts of money on hardware while keeping their services fast and reliable.

Technical Summary: Load Testing for Machine Learning Model Serving Systems at Scale

Problem Statement

Machine learning (ML) model serving has become a dominant consumer of GPU infrastructure, yet capacity planning for these systems remains largely ad hoc. Traditional load testing tools (e.g., JMeter, Locust) and benchmarking suites (e.g., MLPerf) fail to address the unique characteristics of ML inference, leading to two critical issues:

Under-provisioning: Results in Service-Level Objective (SLO) violations, degraded user experience, and production incidents.
Over-provisioning: Wastes substantial GPU resources that could serve other workloads.

The paper identifies four specific challenges that general-purpose tools do not address:

Model Warmup: GPU inference involves JIT compilation, CUDA kernel caching, and weight loading, creating transient performance that does not reflect steady state.
Dynamic Batching: Modern serving systems batch requests, creating a nonlinear relationship between concurrent load and per-request latency.
Architecture Heterogeneity: Systems must handle diverse models (from sub-millisecond lookups to multi-second generative outputs) requiring different testing approaches.
Hardware Sensitivity: Performance varies significantly across GPU generations, memory configurations, and co-location patterns.

Methodology: The Vanguard Framework

The authors present Vanguard, an industrial load testing framework designed to systematically estimate serving capacity through an adaptive, feedback-driven search strategy.

System Architecture

Vanguard is a distributed system comprising ten key components:

Orchestration & Execution: A Task Orchestrator manages the lifecycle of tests, while a Load Replayer generates controlled traffic using an open-loop design to avoid coordinated omission (where processing delays suppress new request generation, masking true latency).
Strategy Engine: Implements the strategy pattern to decouple search algorithms. It supports baseline strategies (Multi-Rate Sweep, Binary Search, Constant-Rate Stability) and the core Adaptive Feedback-Driven Search.
Health Assessment Engine: Maps raw metrics (latency, throughput, error rates, GPU utilization) to five health states (Healthy, Warn, Unhealthy, Critical, Overloaded). It employs multi-metric weighting and hysteresis (requiring states to persist for consecutive windows) to prevent "flapping" near capacity boundaries.
Infrastructure: Includes a Model Registry for versioned artifacts, a Capacity Allocator for resource provisioning, and a Telemetry Logger for reproducibility.

Adaptive Search Strategy

The core contribution is an adaptive algorithm that uses real-time metric feedback to converge on maximum sustainable throughput. It employs three mechanisms:

Dampening: Reduces the step size as the search approaches the capacity boundary to avoid overshooting.
Spike Tolerance: Ignores transient metric spikes below a configurable threshold to prevent premature convergence.
Convergence Detection: Monitors the variance of recent estimates; the search terminates when the coefficient of variation falls below a threshold.

Experimental Design

The study evaluates Vanguard across 14 industrial case studies spanning four architecture classes: Recommendation, Ranking, Vision, and NLP.

Ground Truth: Defined as the P90 Inference-Per-Second (IPS) measured at steady state over a 7-day post-launch window, excluding incident periods and the initial ramp-up.
Variables: The study compares synthetic vs. recorded traffic replay, different hardware (A100 vs. H100), and ablation studies disabling specific ML-aware features (e.g., warmup handling, batching awareness).
Statistical Analysis: Uses non-parametric tests, Bland-Altman analysis for agreement, and multivariate linear regression to identify error factors.

Key Results

1. Reproducibility

Vanguard measurements are highly reproducible on identical hardware (Intra-class Correlation Coefficient = 0.96, Coefficient of Variation = 2.9%). However, cross-hardware tests (A100 vs. H100) show a systematic positive bias of 4.9%, indicating a need for hardware-specific correction factors.

2. Impact of ML-Specific Design Decisions (RQ1)

An ablation study revealed that specific ML-aware features are critical for accuracy:

Workload Calibration: Switching from synthetic traffic to recorded traffic replay reduced estimation error from ~30% to 2–6%. This was the single largest accuracy gain.
Warmup Handling: Properly excluding the transient warmup period reduced error by 22.2%.
Other Factors: GPU health monitoring, multi-metric SLOs, and batching awareness also contributed significant accuracy improvements (7–14% error reduction each).

3. Capacity Prediction Accuracy (RQ2)

Overall Accuracy: Vanguard estimates closely matched observed post-launch capacity with a mean bias of −2.1% (slight underestimation, which is operationally preferable) and 95% limits of agreement of [−14.8%, +10.6%]. The correlation was strong ( $R^2 = 0.94$ ).
Error Predictors: Regression analysis identified Model Size (log-transformed parameter count) and Co-location Density (number of co-tenanted models) as the strongest predictors of estimation error. Larger models exhibit variable warmup behavior, while co-located workloads introduce interference not captured in isolated tests.

Significance and Contributions

The paper claims to be the first empirical study of load testing specifically for ML model serving at scale. Its significance lies in:

Bridging the Gap: It addresses the disconnect between traditional load testing and the specific needs of ML capacity planning, offering a framework that handles dynamic batching, warmup, and hardware sensitivity.
Industrial Impact: In 14 case studies, Vanguard deployment led to:
- A reduction in estimation error from 30% to 2–6%.
- GPU resource reductions of 49–83% for individual models by eliminating over-provisioning.
- A substantial reduction in launch incidents caused by under-provisioning.
Actionable Guidelines: The authors distill six lessons for practitioners, emphasizing that workload representativeness (using recorded traffic) and warmup handling are more impactful than algorithmic sophistication. They also highlight the necessity of hysteresis in health assessment and the importance of accounting for co-location effects.

The paper concludes that while the adaptive strategy is effective for current architectures, future work is needed to address generative models (LLMs), which present distinct challenges regarding variable output lengths and KV-cache pressure.