🤖 machine learning

From Tokens to Watt-hours: Analytical Energy Estimation for LLM Inference on Modern GPUs

This paper introduces an empirically calibrated, analytical methodology for estimating the energy consumption of LLM inference on NVIDIA H100 GPUs by decomposing workloads into compute and memory-traffic components, enabling transparent, reproducible energy assessments without requiring direct hardware measurement.

Original authors: Tina Vartziotis, Rodopi Kosteli, Elli Vartziotis, George Dasoulas, Michael Keckeisen, Konstantinos Skianis, Sotirios Kotsopoulos, Francesca Dominici

Published 2026-07-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tina Vartziotis, Rodopi Kosteli, Elli Vartziotis, George Dasoulas, Michael Keckeisen, Konstantinos Skianis, Sotirios Kotsopoulos, Francesca Dominici

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet as a giant, bustling library where millions of people are constantly asking a magical librarian (an Artificial Intelligence) to write stories, solve math problems, or summarize news. For a long time, we only worried about how much "brainpower" this librarian needed to learn its tricks (training). But now, the librarian is working non-stop, answering millions of questions every day. This constant work uses a massive amount of electricity, which is bad for the planet. The big question for scientists and engineers is: "How much energy does it actually take to answer just one question?" The trouble is, measuring this energy directly is like trying to weigh a single grain of sand while it's being blown by a hurricane; you need special, expensive sensors and perfect conditions that most people don't have. Without a way to guess the energy cost, it's hard to know if one AI is "greener" than another or if a specific way of asking a question is wasting power.

This paper, titled "From Tokens to Watt-hours," offers a clever way to solve that puzzle without needing to plug in a power meter. The authors, a team of researchers from universities and tech companies, have built a mathematical "calculator" that estimates how much energy an AI uses just by looking at the size of the AI and the length of the conversation. Think of it like a nutrition label for AI: instead of counting calories, it counts "Watt-hours" (a unit of energy). They tested their calculator on powerful computer chips called NVIDIA H100s, which are the engines driving many of today's smartest AIs.

Here is how their calculator works, using a simple analogy. Imagine the AI is a chef in a kitchen. To cook a meal (generate an answer), the chef does two things: they read the recipe and gather ingredients (this is the "compute" part), and they run back and forth to the pantry to grab spices and tools (this is the "memory" part). The researchers found that for short orders, the chef mostly just cooks, so the energy cost is mostly about the cooking. But for long, complex orders, the chef has to run to the pantry so many times that the energy spent running around starts to dominate the bill.

The paper breaks down the energy cost into two main phases:

The "Prefill" Phase: This is when the chef reads the customer's order (the input prompt). The energy cost here depends on how long the order is.
The "Decode" Phase: This is when the chef writes the answer, word by word. The energy cost here depends on how long the answer is.

The researchers discovered that the energy cost isn't just about how big the AI is (how many "ingredients" or parameters it has). It also depends heavily on how long the conversation gets. They found that as the answer gets longer, the energy doesn't just go up in a straight line; it curves upward sharply. Why? Because every new word the AI writes requires it to look back at all the previous words to make sense of the story. It's like the chef having to re-read the entire recipe every time they add a new pinch of salt. The longer the recipe, the more running back and forth is needed, and the more energy is wasted.

Using their formula, the team calculated that a small AI (with about 0.3 billion parameters) uses very little energy, roughly 0.001 Watt-hours for a request. But a giant AI (with 120 billion parameters) uses a lot more—about 0.15 Watt-hours for the same task. They also showed that for a 70-billion-parameter model, their estimate was very close (within 5%) to real-world measurements taken by other scientists, proving their "calculator" is reliable enough to use when you can't measure the power directly.

The paper explicitly states that this tool is not a perfect replacement for measuring electricity with a meter. It doesn't count the energy used by the air conditioning, the internet cables, or the other computers in the data center. It only counts the energy used by the specific chip doing the thinking. However, the authors argue that this chip energy is the biggest part of the bill that engineers can control.

So, what does this mean for the future? The paper suggests that if we want to save energy, we shouldn't just try to make bigger AIs. Instead, we should try to make the "recipes" shorter (compressing the input), ask for shorter answers, or use smaller, specialized chefs for simple jobs. The authors hope this calculator will help developers compare different AI models and choose the ones that are more energy-efficient, helping to build a "greener" digital world without needing to install a power meter on every single computer.

Technical Summary: From Tokens to Watt-hours: Analytical Energy Estimation for LLM Inference on Modern GPUs

1. Problem Statement

The operational energy consumption of Large Language Model (LLM) inference is a critical component of the environmental footprint of deployed AI systems. While Green AI research has extensively covered training costs, inference energy—driven by continuous, demand-scaled autoregressive token generation—remains a significant sustainability concern.

Current approaches to estimating AI energy consumption face a dichotomy:

Measurement-based studies rely on hardware telemetry, power instrumentation, or infrastructure-specific monitoring. While valuable, these results are tied to specific hardware, serving systems, and runtime configurations, limiting their applicability in comparative studies, early-stage system design, and sustainability reporting where such instrumentation is unavailable.
Analytical models based on transformer scaling laws and microarchitectural benchmarks provide FLOP counts but often fail to offer request-level or token-level energy estimates that account for memory traffic and hardware-specific coefficients.

This creates a gap for a transparent, reproducible estimator that combines transformer compute modeling, memory-traffic attribution, and hardware-specific energy coefficients to estimate GPU inference energy without requiring runtime instrumentation.

2. Methodology

The authors propose a semi-analytical, empirically calibrated methodology for estimating accelerator-side GPU energy consumption for LLM inference on NVIDIA H100-class accelerators. The framework explicitly separates the prompt prefill phase from the autoregressive decode phase, as these exhibit distinct compute and memory-access patterns.

2.1. Energy Decomposition

The total GPU energy ( $E_{GPU}$ ) for a request is modeled as the sum of tensor-core compute energy ( $E_{compute}$ ) and high-bandwidth memory (HBM) movement energy ( $E_{memory}$ ):
$E_{GPU} = E_{compute} + E_{memory}$

Compute Term: Proportional to the number of tensor-core floating-point operations (FLOPs).
$E_{compute} = \alpha_{TC} C_{TC}$
Where $\alpha_{TC}$ is the hardware-specific energy per FLOP and $C_{TC}$ is the total FLOP count.
Memory Term: Proportional to the number of bits transferred through HBM.
$E_{memory} = e_{HBM} Q_{HBM} \eta(N)$
Where $e_{HBM}$ is the energy per bit, $Q_{HBM}$ is the total traffic in bits, and $\eta(N)$ is a calibrated memory-inefficiency factor.

2.2. Traffic Modeling

The memory traffic ( $Q_{HBM}$ ) is decomposed into three primary components:

Parameter Access: Traffic required to load model weights.
KV-Cache Write: Traffic to store new key-value pairs during generation.
Attention-Related Read: Traffic to read accumulated context from the KV cache.

The model distinguishes between dense transformer operations (scaling linearly with parameters and tokens) and attention corrections (scaling super-linearly with sequence length due to quadratic KV-cache read requirements).

2.3. Calibration and Coefficients

The methodology uses fixed hardware coefficients derived from microarchitectural measurements of GPU tensor-core computation and HBM access (specifically for FP16/BF16 on H100-class GPUs):

$\alpha_{TC} = 0.52$ pJ/FLOP
$e_{HBM} = 11.68$ pJ/bit

To account for non-ideal hardware behavior (e.g., parameter reuse, cache residency, and memory inefficiency), the authors employ calibrated scaling factors ( $\gamma(N)$ , $s_{attn}(N)$ , $\eta(N)$ ) fitted to reported energy measurements from prior literature.

2.4. Simplified Estimator

For scenarios where detailed architecture metadata (e.g., layer count, hidden dimension) is unavailable, a simplified parameter-only estimator is defined. It approximates output-token energy as linearly proportional to the parameter count ( $N$ ) and uses a prompt-length-dependent multiplier for input tokens.

3. Key Contributions

Analytical Framework: A structured methodology to estimate request-level and normalized token-level GPU inference energy without direct runtime measurement.
Phase Separation: Explicit decomposition of energy into prompt prefill and autoregressive decode phases, enabling granular analysis of input vs. output token costs.
Component Breakdown: Decomposition of total energy into compute, parameter-access, KV-cache write, and attention-read components, revealing the scaling behavior of each.
Empirical Calibration: Integration of hardware-specific energy coefficients with empirically calibrated memory-traffic factors to bridge the gap between theoretical FLOP counts and actual energy consumption.
Actionable Insights: Identification of specific mechanisms for reducing energy, including model size reduction, shorter generations, KV-cache quantization, prompt compression, and improved batching.

4. Results and Evaluation

The estimator was evaluated against a set of transformer-based LLMs (ranging from <3B to >120B parameters) and compared against measurement-based studies.

Scaling with Sequence Length: Compute energy and KV-cache write traffic scale linearly with the number of generated tokens ( $T_{out}$ ). However, attention-related memory traffic exhibits super-linear (quadratic) growth, becoming a dominant energy contributor as sequence lengths increase.
Scaling with Model Size: Total request energy scales approximately linearly with model size ( $N$ ) on a log-log scale. However, memory energy shows non-monotonic behavior across different models due to architectural variations (e.g., hidden dimension and layer count) that do not scale uniformly with parameter count.
Energy Decomposition: For short to moderate sequence lengths, compute energy dominates. As sequence length increases, the attention-related memory component grows rapidly, causing an upward curvature in the total energy trend.
Accuracy: When compared to measurement-based results for a 500-token input/500-token output workload, the analytical estimates matched measured values within 4.73% to 27.23%. The lowest error (4.73%) was observed for the 70B parameter model.
Token-Level Estimates: The study provides simplified token-level energy estimates (e.g., ~218 mJ/output token for a 70B model), demonstrating the linear dependence of energy on parameter count.

5. Significance and Claims

The paper positions this methodology not as a replacement for physical power measurements, but as a transparent, reproducible, and assumption-explicit approximation.

Scope: The estimates are strictly limited to accelerator-side operational energy (GPU compute and HBM movement). They explicitly exclude system-level overheads such as CPU execution, host memory, networking, storage, cooling, and Power Usage Effectiveness (PUE).
Utility: The framework is designed for:
- Model Comparison: Comparing energy efficiency across heterogeneous model inventories.
- Green-Coding Analysis: Assessing the impact of architectural choices and interventions.
- Design-Time Evaluation: Making deployment decisions when full datacenter-level instrumentation is unavailable.
Limitations: The authors acknowledge that the model abstracts away specific effects like tensor-parallel communication, kernel fusion, batching policies, and specific inference engine implementations. Consequently, the factors should be recalibrated for different GPU generations or serving configurations.

In conclusion, the paper provides a foundational tool for understanding the energy scaling of LLM inference, highlighting that while compute is dominant for short sequences, memory traffic (specifically attention reads) becomes a critical bottleneck for long-context generation.