Workload composition smooths aggregate power demand while sustaining short-horizon ramps in AI data centers

1. Problem Statement

Artificial Intelligence (AI) is driving a massive surge in electricity demand, yet the grid-facing power dynamics of AI data centers remain poorly understood. Current planning models often treat data centers as static, exogenous hourly loads based on annual energy consumption or aggregate GPU counts. This approach fails to capture critical short-horizon dynamics (minutes to hours), such as load ramping and variability, which are crucial for grid reliability, reserve requirements, and emissions accounting.

A specific gap exists in understanding hybrid AI data centers, where latency-sensitive inference workloads and queue-managed batch workloads coexist on shared GPU pools. It is unclear how the interaction between these two distinct workload types affects aggregate power demand, specifically whether they act as simple weighted averages or exhibit complex, non-linear behaviors.

2. Methodology

The authors developed a trace-calibrated simulation framework to model a hybrid AI data center. The model integrates three empirical data sources:

1. Batch Workloads: 98,000 batch jobs with per-job multi-GPU power profiles from MIT's High-Performance Computing (HPC) facility.
2. Inference Workloads: 44 million request-level traces (code and conversation) from Microsoft Azure's Large Language Model (LLM) services.
3. Power Mapping: Serving measurements mapping inference token throughput to GPU demand and power (based on vLLM-style continuous batching).

Key Modeling Assumptions & Mechanics:

• Shared-GPU Policy: Latency-sensitive inference receives priority access to GPUs. Batch jobs run on the residual capacity.
• Scheduling:
  • Inference: Modeled with a continuous-batching approximation; no persistent internal queue (requests are served immediately or rejected/rerouted).
  • Batch: Modeled as checkpointable jobs in a discrete-event simulator. Jobs wait in queues and can be interrupted/resumed. Two scheduling rules were tested: First-Come-First-Served (FCFS) with backfilling and Smallest-Work-First (SWF).
• Experimental Design: The study isolates the effect of workload composition (the ratio of inference to batch work) while holding total GPU capacity and system utilization fixed.
• Metrics:
  • Aggregate Variability: Measured by the Coefficient of Variation (COV) of total power.
  • Short-Horizon Ramping: Measured by the median 15-minute ramp rate (change in power over time).

3. Key Contributions

The paper challenges the assumption that hybrid systems are linear combinations of their parts. It introduces the concept that workload composition non-linearly reshapes grid impact, decoupling aggregate smoothness from short-term ramping.

Core Findings:

1. U-Shaped Variability: As the share of inference work increases, aggregate power variability follows a U-shaped curve.

   • At low inference shares (mostly batch), variability is moderate.
   • At intermediate mixes, variability reaches a minimum.
   • At high inference shares, variability rises again.
   • Mechanism: Queued batch jobs fill "valleys" in capacity left by fluctuating inference demand, smoothing the aggregate load. This effect is strongest at intermediate mixes and higher utilizations.

2. Hump-Shaped Ramping: In contrast, short-horizon ramping follows a hump-shaped curve (inverted U) at higher utilizations.

   • Ramping increases as inference share rises because inference fluctuations propagate directly to power.
   • However, at very high inference shares, ramping may flatten or decline due to capacity saturation (the system cannot ramp up further).
   • Mechanism: Batch jobs buffer burstiness via queues, but inference transmits fluctuations directly. While batch activity offsets some inference ramps, the offset is incomplete.

3. Asymmetric Propagation:

   • Batch: Burstiness is absorbed by the queue. The link between job arrivals and realized power is weak initially but tightens once jobs are running.
   • Inference: Fluctuations in arrivals are transmitted directly to realized power. Smoothing only occurs via capacity saturation or longer request residence times (e.g., slower token generation).

4. Results

• Decoupling of Metrics: Aggregate smoothness (variability) and short-horizon ramping are distinct properties. A workload mix that minimizes variability does not necessarily minimize ramping.
• Optimal Smoothing: The most "grid-friendly" mix for variability occurs at intermediate inference shares (where batch fills inference gaps). However, this same mix may still impose significant ramping stress on the grid due to the direct nature of inference fluctuations.
• Utilization Dependence: The U-shape in variability and the hump-shape in ramping become more pronounced at higher system utilizations.
• Checkpointing Impact: Shorter checkpoint intervals allow batch jobs to be more flexible, shifting the minimum variability point and improving smoothing. Longer intervals reduce this flexibility.

5. Significance

• Grid Planning: The findings argue against modeling AI data centers as static loads. Grid planners must account for the internal dynamics of workload composition. A data center with a "smooth" aggregate load profile might still pose significant challenges due to high short-horizon ramping.
• Operational Strategy: Data center operators can potentially optimize grid impact by managing the mix of batch and inference workloads, though they must balance variability reduction against ramping risks.
• Policy & Emissions: Accurate emissions accounting and flexibility valuation require understanding these non-linear dynamics, as the timing of power consumption (ramping) is as critical as the total volume.
• Future Research: The study highlights the need for models that move beyond annual energy totals to capture the temporal resolution of digital service dynamics translating into electrical demand.

In summary, the paper demonstrates that hybrid AI data centers are dynamic systems where the interplay between queued batch jobs and immediate inference requests creates unique, non-additive power profiles. This complexity necessitates a shift in how the grid integrates and manages AI infrastructure.