🤖 machine learning

WattGPU: Predicting Inference Power and Latency on Unseen GPUs and LLMs

WattGPU introduces a novel framework that accurately predicts the power consumption and inference latency of Large Language Models on unseen GPUs and LLMs using only public metadata, significantly outperforming traditional physical baselines without requiring hardware profiling.

Original authors: Mauricio Fadel Argerich, Jonathan Fürst, Marta Patiño-Martínez

Published 2026-07-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mauricio Fadel Argerich, Jonathan Fürst, Marta Patiño-Martínez

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to rent a car for a road trip, but you don't know which car is the most fuel-efficient for your specific luggage and route. Usually, to find out, you'd have to physically drive every single car on the market with your exact luggage, measure the gas, and time the trip. That would take forever and cost a fortune.

This paper introduces WattGPU, a "crystal ball" for computer chips (GPUs) and AI models (LLMs). Instead of driving every car, WattGPU predicts exactly how much energy a specific AI model will use and how fast it will run on a specific computer chip, just by looking at their spec sheets.

Here is the breakdown of how it works, using simple analogies:

The Problem: The "Guessing Game"

Big companies and small businesses are using AI chatbots more and more. These chatbots need powerful computer chips (GPUs) to run.

The Issue: Not all chips are created equal. Running a small AI model on a massive, powerful chip is like putting a bicycle tire on a semi-truck—it wastes a huge amount of fuel (electricity).
The Old Way: To find the best match, operators had to physically test every combination of AI model and computer chip. This is expensive, slow, and requires owning all the hardware.
The Gap: Existing tools could guess, but they failed when they encountered a new chip or a new AI model they hadn't seen before.

The Solution: WattGPU (The "Spec Sheet Oracle")

The authors built two smart prediction models (like a very advanced calculator) that only need public information:

The AI's "Resume": How big is it? How many layers does it have? (Metadata from Hugging Face).
The Chip's "Spec Sheet": How fast is it? How much memory does it have? What is its maximum power limit? (Manufacturer specs).

The Magic Trick:
The system learns the "personality" of different chips and AI models. Once trained, it can look at a brand-new chip it has never seen and a brand-new AI model it has never met, and accurately guess:

Power Draw: How many watts of electricity will it suck up?
Latency (ITL): How long will it take to generate one word (token) of text?

The Analogy: The Restaurant Kitchen

Think of the AI model as a recipe and the GPU as a kitchen.

Some recipes are simple (making toast); some are complex (a 10-course banquet).
Some kitchens have tiny stoves; others have industrial ovens.

Old Method: You have to cook the recipe in every kitchen to see how much gas it uses and how long it takes.
WattGPU Method: You look at the recipe's ingredient list and the kitchen's stove size. The system predicts: "If you cook this specific recipe in that specific kitchen, it will use 13% less gas than you think, and it will take 5 seconds per dish."

What Did They Prove?

The researchers tested WattGPU on a massive dataset of 42 different AI models and 8 different server-grade computer chips. They used a strict testing method called "Leave-One-Out," which is like taking a test where you are forbidden from studying the specific question you are about to answer.

The Results:
- Power Prediction: It was incredibly accurate. For offline tasks (batch processing), it was off by less than 3.4%. For server tasks (real-time chat), it was off by less than 13.5%.
- Speed Prediction: For real-time chat, it was off by less than 8.5%.
- Ranking: It was excellent at telling you which chip is better than another, even if the exact numbers weren't perfect.

Why This Matters (The "Aha!" Moment)

The paper highlights a surprising finding using a specific example: Llama 3.1 8B.

If you just look at the "fuel efficiency rating" (TDP) of a chip, you might think a small, low-power chip (L4) is the best choice.
However, WattGPU predicted that this small chip would be too slow to meet the speed requirements.
The "obvious" high-power chip (H100) was fast but wasted 43% more energy than a medium-sized chip (A30) that was just as fast.
The Lesson: WattGPU found the "Goldilocks" chip—the one that is fast enough but doesn't waste energy. Without this tool, people would have picked the wrong chip and wasted massive amounts of electricity and money.

What It Doesn't Do (The Limits)

The paper is honest about what WattGPU cannot do yet:

It works best on standard, "dense" AI models. It doesn't yet handle complex "Mixture of Experts" models (which are like a team of specialists working together) or models that have been heavily compressed (quantized).
It struggles a bit more with "offline" batch processing speeds compared to real-time chat speeds, because the software behaves differently when processing huge batches at once.

The Bottom Line

WattGPU is a tool that lets you skip the physical testing. By using only public specs, it helps operators choose the right computer chip for their AI, saving money and electricity without needing to build a lab full of hardware to test every possibility. It turns the "guessing game" of AI deployment into a calculated prediction.

Technical Summary: WattGPU

Problem Statement
The rapid proliferation of Large Language Model (LLM) inference workloads, particularly in small-to-medium enterprise deployments, has created a significant and growing energy footprint. Optimizing these deployments requires matching specific LLMs to the most energy-efficient GPUs. However, operators currently lack tools to perform this matching without exhaustively profiling every potential LLM-GPU combination on physical hardware. This profiling process is prohibitively expensive, requiring physical access to diverse hardware, specialized expertise, and significant resources. Existing predictive models either rely on profiling data or fail to generalize to hardware unseen during training, leaving a gap in the ability to estimate power draw and latency for unseen configurations using only public metadata.

Methodology
To address this, the authors introduce WattGPU, a framework featuring two predictive models for Mean GPU Power Draw ( $\bar{P}$ ) and Inter-Token Latency (ITL). The core innovation is that these models require zero hardware access or profiling once trained; they rely exclusively on publicly available LLM metadata (e.g., parameter count, layers, attention heads from Hugging Face) and GPU manufacturer specifications (e.g., TDP, memory bandwidth, clock speeds).

Feature Engineering: The authors derived features to capture hardware dynamics, including a Boosting ratio (frequency scaling), Bandwidth Latency (time to read weights), and Compute Latency (theoretical lower bound for token computation).
Model Architecture: The authors evaluated various algorithms and selected XGBoost (Gradient-Boosted Regression Trees) as the optimal model.
- For power prediction, the target variable is normalized by the GPU's Thermal Design Power (TDP) to learn scale-invariant behavior, facilitating generalization to unseen GPUs.
- Hyperparameters were tuned via grid search, with specific configurations for power (max_depth=6, n_estimators=100) and latency (max_depth=5, n_estimators=200).
Evaluation Strategy: The models were evaluated on a dataset comprising 42 open-source LLMs (0.1B–27B parameters) and 8 server-grade NVIDIA GPUs (ranging from V100 to H200). Evaluation utilized rigorous cross-validation strategies:
1. 5-fold Cross-Validation (CV): Generalization to unseen (GPU, LLM) combinations of known hardware/models.
2. Leave-One-GPU-Out (LOGO): Generalization to completely unseen GPUs.
3. Leave-One-LLM-Out (LOLO): Generalization to completely unseen LLMs.
  Scenarios included Offline (maximum throughput) and Server (Poisson-distributed requests with latency constraints).

Key Results
The WattGPU models demonstrate significant improvements over analytical baselines (Load-Scaled TDP for power and Roofline for latency), particularly in generalization scenarios.

Mean Power Draw ( $\bar{P}$ ):
- Unseen GPUs (LOGO): Achieved a Median Absolute Percentage Error (MdAPE) of ≤3.4% for offline scenarios and ≤13.5% for server scenarios.
- Ranking: Maintained strong GPU ranking correlations for server scenarios (Kendall $\tau \ge 0.76$ ).
- Baseline Comparison: Reduced MdAPE by approximately 2× for completely unseen GPUs and 4× for unseen LLM-GPU combinations in server scenarios compared to the strongest baselines.
Inter-Token Latency (ITL):
- Server Scenarios: Achieved an MdAPE of ≤8.5% on unseen GPUs (LOGO).
- Offline Scenarios: Performance was more challenging, with an MdAPE of 24.9% on unseen GPUs, though the model still outperformed the Roofline baseline in absolute error (24.9% vs. 80.2% for LOGO).
- Ranking: The models achieved high ranking quality, with Kendall $\tau \ge 0.76$ for server scenarios.

Significance and Claims
The paper claims to present the first predictive models for LLM inference power and latency evaluated specifically on unseen GPUs. Its primary significance lies in enabling energy-aware deployment decisions without the need for physical benchmarking.

Practical Impact: The authors illustrate that standard heuristics (like TDP or TFLOPS/Watt) can be misleading. For example, selecting an H100 over an A30 for a low-load Llama 3.1 8B deployment might seem efficient based on TFLOPS/Watt, but WattGPU predicts it would draw 43% more power while meeting the same latency SLA.
Sustainability: By allowing operators to select the most efficient GPU for a specific LLM and workload scenario, the tool directly contributes to reducing operational costs and carbon emissions, aligning with sustainability goals and emerging regulations like the EU AI Act.
Limitations and Future Work: The authors acknowledge that ITL prediction in offline scenarios remains difficult (MdAPE 24.9%) because static features cannot fully capture dynamic batching behaviors (e.g., vLLM's continuous batching). Future work aims to incorporate workload-aware features and extend the approach to Mixture-of-Experts (MoE) and quantized models. The current evaluation is limited to dense models on single-GPU configurations.