Breaking the Ice: Analyzing Cold Start Latency in vLLM

Technical Summary: Breaking the Ice: Analyzing Cold Start Latency in vLLM

Problem Statement
As Large Language Models (LLMs) are deployed at scale, particularly within serverless computing paradigms, the "cold start" latency of inference engines has become a critical bottleneck. In serverless environments, cloud providers spin up new container instances to handle traffic spikes, introducing latency orders of magnitude higher than serving requests on "warm" instances. This latency primarily impacts the Time-to-First-Token (TTFT). While vLLM has emerged as the de-facto inference engine for many workloads due to its optimizations (e.g., PagedAttention, chunked prefill), its startup behavior lacks a systematic, end-to-end characterization. The rapid evolution of vLLM (e.g., the introduction of the V1 API and torch.compile) and its increasing codebase complexity have made it difficult to understand how architectural changes impact startup performance. Existing literature often focuses on isolated components (e.g., checkpoint loading or state materialization) rather than the holistic startup process, hindering the design of efficient autoscaling policies for containerized LLM services.

Methodology
The authors conduct a comprehensive performance characterization of the vLLM startup process, defining "startup time" as the duration from the initialization of the inference engine to the point where it is fully operational and ready to serve requests.

• Experimental Setup: Experiments were conducted using vLLM v0.10.1.1 across 22 diverse LLMs (ranging from 0.5B to 32B parameters, including MoE architectures) on NVIDIA H100 and L40S GPUs. The study utilized four distinct hardware nodes with varying CPUs (AMD EPYC, Intel Xeon) and storage configurations (DRAM cache vs. PCIe 5.0 SSDs).
• Decomposition: The startup process was decomposed into six foundational steps:
  1. Framework Bootstrapping (platform detection, imports, worker initialization).
  2. Tokenizer Initialization.
  3. Model Loading (structure initialization and weight loading).
  4. Torch Compilation (Dynamo transformation and loading compiled graphs).
  5. KV Cache Profiling.
  6. CUDA Graph Capturing.
• Analysis: The authors measured the latency of each step under various conditions, analyzing scaling trends with respect to model parameters (size, layers, vocabulary), system parameters (CPU/GPU types), and configuration flags. They also evaluated the impact of different weight loading backends (Safetensors, Run:ai Streamer, CoreWeave Tensorizer) and the presence of cached compiled graphs.
• Predictive Modeling: Based on the observed scaling trends, the authors developed a lightweight, white-box analytical predictor. This model uses linear regression for each of the six steps independently, trained on model configurations and environment characteristics, to predict total startup latency.

Key Results

• CPU-Bound Nature: The analysis reveals that the vLLM startup process is predominantly CPU-bound. While the final two steps (KV Cache Profiling and CUDA Graph Capturing) are GPU-bound, the majority of the latency (including framework bootstrapping, tokenization, model loading, and compilation) is driven by CPU performance. Changing the GPU (H100 vs. L40S) yielded negligible speedups for most steps, whereas changing the CPU resulted in significant performance variations.
• Scaling Trends:
  • Tokenizer Initialization: Scales linearly with tokenizer file size (PCC = 0.99).
  • Model Loading: Weight loading time scales linearly with model parameter size and precision (PCC = 1.00). Loading from SSDs slows this step by ~0.5x but has minimal impact on the total startup time (which improves by only ~1.04x) because weight loading constitutes a small fraction of the total duration.
  • Torch Compilation: Dynamo transformation time scales with the number of layers and layer complexity (approximated by compiled graph file size, PCC = 0.96). Loading compiled graphs also scales linearly with graph size (PCC = 0.95).
  • KV Cache Profiling: Scales linearly with model parameter size for non-MoE models (PCC = 0.92). MoE models deviate due to dynamic expert routing.
  • CUDA Graph Capturing: Scales linearly with both model size and the number of batch sizes captured (PCC = 1.00).
• Impact of Caching: Disabling the cache for compiled graphs increases the "storing compiled graphs" step from ~3–6 seconds to ~11–21 seconds, highlighting the importance of caching for subsequent runs.
• Predictor Accuracy: The proposed analytical predictor achieves high accuracy with a Mean Squared Error (MSE) of 2.42 seconds and a maximum error of 2.08 seconds on validation models. The model remains accurate (MSE 2.62s) when validated on vLLM v0.11, demonstrating its robustness to version updates.

Significance and Claims
The paper claims to provide the first detailed, systematic characterization of the vLLM startup process. Its primary contributions are:

1. Decomposition: Identifying and isolating six foundational steps that constitute the startup process, revealing that the process is largely CPU-bound rather than GPU-bound.
2. Interpretability: Establishing consistent, interpretable scaling trends (mostly linear) between specific parameters (e.g., model size, tokenizer size, graph size) and latency for each step.
3. Predictive Capability: Developing a lightweight, white-box analytical model that accurately predicts startup latency for given hardware and model configurations. This tool is designed to assist in resource planning and autoscaling decisions for serverless LLM environments.
4. Longevity: The authors argue that while specific parameter values may change with vLLM updates, the six-step decomposition captures fundamental operations inherent to modern inference engines. Therefore, the methodology remains applicable, requiring only parameter re-tuning for future versions rather than a complete re-analysis.

The paper positions this work as a foundational step toward designing impactful autoscaler policies for containerized, serverless inference frameworks, moving beyond treating startup as a monolithic black box.