🤖 machine learning

The Model Parking Tax: Quantifying the Hidden Energy Cost of Always-On GPU Model Deployment

This paper empirically demonstrates that the energy cost of keeping AI models loaded in GPU memory is primarily driven by a discrete power increase from the CUDA context's DVFS transition rather than VRAM occupancy, revealing that the energy-optimal strategy for model deployment depends on request arrival rates and cold-start latency rather than model size.

Original authors: Sai Sathvik Vadari

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sai Sathvik Vadari

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Parking Tax" on AI

Imagine you run a busy restaurant. You have a rule: "Once a table is set for a VIP guest, we keep it set forever, even if no one is sitting there."

Why? Because if a new guest arrives, you don't want to waste time setting the table (silverware, napkins, menu). You want them to sit down and eat immediately.

In the world of AI, companies do the same thing with powerful computer chips called GPUs. When they load an AI model (like a chatbot) onto a GPU, they leave it there, running 24/7, even when no one is asking it questions. They do this to avoid the "cold start" delay—the time it takes to load the model from scratch.

The Problem: This paper argues that keeping these AI models "parked" on the GPU is incredibly wasteful with electricity, and for a reason nobody expected.

The Hidden Cost: It's Not About the "Size" of the Car

Most people assume that keeping a big AI model (like a 70-billion-parameter monster) on a GPU uses way more electricity than keeping a tiny one (like a 7-billion-parameter model) because the big one takes up more space in the GPU's memory (VRAM).

The paper's big discovery: This is false.

The authors measured this across three different types of high-end GPUs (H100, A100, and L40S). They found that the electricity cost of "parking" a model is almost entirely determined by turning the engine on, not by how much space the car takes up in the garage.

The Analogy: The Idling Car

Think of the GPU like a car engine:

Bare Idle: The car is off. The engine is cold. It uses almost no gas.
The "Context" (The Engine On): As soon as you turn the key to start the car (create a "CUDA context"), the engine revs up to a high speed to be ready to drive instantly.
- The Shocking Part: Once that engine is revving, it doesn't matter if you put a tiny bicycle in the trunk or a massive truck. The engine is still revving at the same high speed, burning the same amount of gas.

The paper calls this the "Model Parking Tax."

The Tax: A fixed amount of electricity (between 26 and 66 Watts, depending on the chip) that you pay the moment you load any model.
The Surprise: Adding more memory to hold a bigger model adds almost zero extra cost. Whether you park a 1GB model or a 64GB model, the electricity bill is the same.

How They Found This

The researchers didn't just guess; they did two things:

Real-World Spying: They watched 14 real GPUs in a data center for 18 days, taking hundreds of thousands of measurements. They saw that when a model was loaded, power jumped up, but changing the model size didn't change the power.
Controlled Experiments: They took three different types of GPUs and systematically filled their memory from empty to full, like pouring water into a bucket. They measured the power at every step.
- Result: The power line was flat. Pouring more "water" (memory) into the bucket didn't make the "engine" work harder.

The "Breakeven" Moment

So, should we turn the engine off to save gas? The paper says yes, but only if you wait long enough.

They calculated a "Breakeven Point." This is the amount of time you have to wait before it becomes cheaper to turn the engine off and restart it later, rather than letting it idle.

The Math: For most modern setups, if you aren't getting a request for 1 to 5 minutes, it is actually cheaper to turn the model off and load it again when someone asks.
The Catch: Small models are the worst offenders. They load in seconds, so they should be turned off immediately after use. But big models take longer to load, so you might keep them on a bit longer. However, the paper notes that the "tax" is the same for both, so small models are the most wasteful if left on.

The Solution: A Smarter Scheduler

The authors built a simple "smart parking meter" (a scheduler). Instead of keeping models on forever, this system checks: "Has it been more than 5 minutes since the last request?"

If Yes: Turn it off.
If No: Keep it on.

When they tested this, they saved 8% to 23% of the electricity compared to the standard "always-on" method, with almost no noticeable delay for the user.

The Bottom Line

The Myth: "We must keep big AI models on the GPU 24/7 because they take too long to load."
The Reality: The cost of keeping them on is a fixed "parking fee" caused by the GPU's engine revving up. The size of the model doesn't matter.
The Fix: We should turn these models off much more often. If a model hasn't been used in a few minutes, turn it off. It saves a massive amount of energy (potentially hundreds of gigawatt-hours a year across the industry) without hurting performance.

In short: You don't need to keep the engine running just because you have a big trunk. If you aren't driving, turn the car off.

Technical Summary: The Model Parking Tax

Problem Statement

The prevailing operating model for AI inference involves deploying models to GPUs and keeping them loaded in VRAM indefinitely to avoid the latency associated with "cold starts." While the latency benefits of this "keep-warm" strategy are well-documented, the ongoing energy cost of readiness has been treated as an unquantified constant. Industry reports estimate GPU idle power at roughly 20% of Thermal Design Power (TDP), but the marginal energy cost of allocating specific amounts of VRAM has never been empirically decomposed.

The core question addressed is whether storing model weights in GPU VRAM incurs a measurable marginal idle energy cost, and if this cost varies across different GPU architectures and memory technologies. Current literature suggests that CUDA contexts force GPUs into higher clock frequencies even when idle, but it remains unclear if the size of the model (VRAM occupancy) further increases power consumption beyond the cost of simply maintaining the context.

Methodology

The authors employ a two-phase approach combining large-scale production telemetry with controlled dose-response experiments across three distinct GPU architectures: NVIDIA H100 (HBM3), A100 (HBM2e), and L40S (GDDR6).

Phase 1: Production Telemetry

Scope: 18 days of data from 14 NVIDIA H100 GPUs across two Kubernetes nodes.
Data Volume: 335,267 idle-state samples (0% utilization) with VRAM allocations ranging from 3 MB to 79 GB.
Instrumentation: Metrics were collected via NVIDIA Data Center GPU Manager (DCGM) Exporter at 30-second intervals.

Phase 2: Controlled Experiments

Design: Within-subject dose-response studies on one GPU of each architecture (H100, A100, L40S).
Protocol:
1. Record "bare idle" power (no CUDA context).
2. Create a persistent CUDA context.
3. Sequentially allocate increasing VRAM (0 to 40–72 GB) using torch.empty tensors.
4. Record power via nvidia-smi for 20 minutes per allocation level.
Validation: The experiments were validated using a real HuggingFace model (Qwen2.5-7B) to ensure that real model weights and CUDA kernels did not alter the idle power profile compared to empty tensors.
Statistical Rigor: The study utilizes Two One-Sided Tests (TOST) to establish equivalence, testing whether the marginal power coefficient ( $\beta$ ) is practically zero (bounded by $\pm 0.1$ W/GB).

Key Findings

1. The "Parking Tax" is a Step Function, Not a Gradient

The study identifies a "Model Parking Tax" composed of two distinct components:

The CUDA Context Overhead ( $\Delta P_{DVFS}$ ): Creating a CUDA context triggers a discrete Dynamic Voltage and Frequency Scaling (DVFS) transition. The Streaming Multiprocessor (SM) clocks jump from low-power idle states (210–345 MHz) to maximum boost frequencies (1410–2520 MHz), regardless of utilization.
- H100: +49.9 W overhead.
- A100: +26.3 W overhead.
- L40S: +66.4 W overhead.
  This step accounts for >98% of the total idle power cost on all tested architectures.
The Marginal VRAM Effect ( $\beta$ ): Once a CUDA context is active, varying the allocated VRAM from 0 GB to the maximum tested capacity (up to 72 GB) results in a power change of less than 1 W.
- The regression slope $\beta$ was found to be statistically indistinguishable from zero on H100 and L40S.
- On the A100, a statistically significant negative slope was observed, but the authors attribute this to thermal drift (temperature decrease over time) rather than a memory physics effect, noting the magnitude is negligible (<0.1 W total).
- Conclusion: $\frac{\partial P}{\partial C} \gg \frac{\partial P}{\partial V}$ . The cost is determined by the presence of a context, not the occupancy of memory.

2. Cross-Architecture Consistency

The finding that idle power is piecewise constant holds across three fundamentally different memory technologies (HBM3, HBM2e, GDDR6) and microarchitectures (Hopper, Ampere, Ada Lovelace). This consistency supports the theoretical prediction that at idle, memory power is dominated by background and refresh circuits (which sweep all rows regardless of data content), making power occupancy-independent.

3. Cold-Start Breakeven Model

The authors derive a breakeven model ( $T^*$ ) to determine the optimal time to evict a model versus keeping it warm:
$T^* = \frac{P_{load} \cdot t_{load}}{P_{park}}$
Where $P_{park}$ is the parking tax (context overhead), $P_{load}$ is the power during loading, and $t_{load}$ is the loading duration.

Result: Breakeven intervals are short, typically 1–5 minutes, depending on the architecture and loading method.
Implication: For small models that load in seconds, the energy cost of keeping them warm often exceeds the cost of a cold start, even with moderate request rates. The breakeven point depends on loading latency and power, not model size.

Significance and Claims

The paper claims to provide the first cross-architecture measurement of idle GPU power as a function of VRAM allocation. Its primary contributions are:

Decomposition of Idle Cost: It empirically separates idle power into a massive, fixed DVFS cost (triggered by the CUDA context) and a negligible, practically zero marginal VRAM cost.
Challenge to Current Practices: It challenges the industry assumption that larger models incur proportionally higher idle energy costs. The paper asserts that a 1 GB model and a 64 GB model pay the "same parking tax" because the cost is driven by the context, not the memory footprint.
Operational Guidance: The authors propose that "size-aware scheduling" (cold-starting small models) and "breakeven-aware eviction" are more energy-efficient than always-on deployment.
Industry Impact Estimation: Based on 2023 shipment data and a 65% utilization rate, the authors estimate that the CUDA context overhead alone accounts for 92–1,745 GWh/year globally, with a base estimate of 462 GWh/year (approx. 180 kT CO2).

The paper concludes that GPU vendors should investigate idle states that decouple clock frequency from context presence, and serverless platforms should prioritize eliminating CUDA contexts over reducing model footprints to achieve energy savings. A proof-of-concept scheduler using the derived breakeven model demonstrated 8–23% energy savings over 24-hour simulations compared to always-on baselines.