💻 computer science

This Is Taking Too Long -- Investigating Time as a Proxy for Energy Consumption of LLMs

This paper proposes using inference time as a proxy to estimate the opaque energy consumption of API-based Large Language Models, validating the approach by correlating time measurements with actual energy data from locally hosted equivalents to help end users understand environmental costs.

Original authors: Lars Krupp, Daniel Geißler, Francisco M. Calatrava-Nicolas, Vishal Banwari, Paul Lukowicz, Jakob Karolus

Published 2026-03-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lars Krupp, Daniel Geißler, Francisco M. Calatrava-Nicolas, Vishal Banwari, Paul Lukowicz, Jakob Karolus

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Black Box" Energy Bill

Imagine you order a meal at a restaurant. The menu tells you the price of the food, but it doesn't tell you how much electricity the chef used to cook it, how much water was wasted washing the dishes, or how much fuel the delivery truck burned to get the food to your table.

In the world of Artificial Intelligence (AI), specifically Large Language Models (LLMs) like the ones powering ChatGPT or Mistral, this is exactly what's happening. When you ask an AI a question, you get an answer, but you have no idea how much energy it took to generate that answer. The companies running these models treat their servers like a "black box"—they don't share the energy bill.

This is a problem because AI is getting huge, and it's eating up massive amounts of electricity, which hurts the environment. We need a way to know how "dirty" our AI usage is, but we can't plug a meter into a company's secret server farm.

The Detective's Trick: Time is the Clue

Since the researchers (Lars Krupp and his team) couldn't peek inside the secret server, they decided to use a clever workaround: Time.

Think of it like this: If you know how fast a specific car engine burns gas, and you know how long a trip took, you can guess how much fuel was used.

The Hypothesis: The time it takes for an AI to answer a question is directly linked to how much energy it uses.
The Catch: A fast answer could mean a super-efficient engine, or it could mean a massive, gas-guzzling engine that just happened to be empty of traffic. To solve this, the researchers needed a "control group."

The Experiment: The "Local vs. Remote" Race

To test their theory, the team set up a race between two runners:

The Local Runner (The Control): They installed the AI models on their own computers in Germany using specific, known graphics cards (GPUs). They knew exactly how much electricity these cards used. They ran the same "tasks" (like writing a math explanation or a creative story) over and over again.
The Remote Runner (The Mystery): They asked the same questions to the AI models hosted on the internet (via API). They couldn't see the hardware, but they could measure exactly how long it took to get the answer.

The Analogy: Imagine you are trying to guess what kind of car a stranger is driving. You can't see the car, but you can time how long it takes them to drive from Point A to Point B.

If they are very fast, you might guess they have a Ferrari (H100 GPU).
If they are slower, you might guess they have a standard sedan (A100 GPU).
By comparing their time to the times of cars you know you own (your local tests), you can make a very good guess about what they are driving.

What They Found

The results were quite revealing:

The Internet is Faster (But maybe "heavier"): The AI models on the internet were significantly faster than the ones running on the researchers' local computers.
The "GPU Detective" Work: By looking at how fast the internet AI was, the researchers could deduce that the companies were likely using the newest, most powerful graphics cards (the H100 and H200 chips). These are the "Ferraris" of the AI world.
The Energy Estimate: Once they figured out what hardware was likely being used, they could calculate the energy cost.
- They found that using the "Time" method gave them a pretty accurate estimate of the energy consumption.
- They also discovered that the "Free" and "Paid" versions of the AI models ran on very similar hardware, meaning the free version isn't necessarily running on "worse" (slower) equipment; it's just a similar setup.

Why This Matters

This paper is like giving everyone a fuel gauge for the AI they use every day.

For Regular People: It helps us understand that every time we ask an AI to write an email or solve a math problem, there is a hidden "carbon cost" attached to it.
For Companies: It puts pressure on AI providers to be more transparent. If we can estimate their energy use just by timing their responses, they can't easily hide the environmental impact.
For the Planet: It helps researchers and policymakers figure out how to make AI more sustainable. If we know which setups use the most energy, we can try to optimize them.

The Bottom Line

The researchers proved that time is a good proxy for energy. Even though we can't see the servers, by simply timing how long an AI takes to think, we can make a smart guess about the hardware behind it and, consequently, how much electricity it is burning. It's a simple, low-tech way to shine a light on a very high-tech, opaque problem.

1. Problem Statement

The energy consumption of Large Language Models (LLMs) poses significant environmental and sustainability challenges. However, for users accessing these models via APIs (e.g., Mistral, OpenAI), energy costs remain opaque. Providers rarely disclose hardware specifications, model configurations, or actual power draw, treating their infrastructure as a "black box."

The Gap: Researchers and users lack direct access to weights, inference configurations, or hardware details, making robust energy estimation impossible.
The Opportunity: While energy data is hidden, inference time (task completion time) is a directly measurable metric. The authors investigate whether inference time can serve as a reliable proxy to estimate the energy consumption of API-based LLMs.

2. Methodology

The authors propose a methodology that correlates inference time with energy consumption by grounding API-based measurements against locally hosted equivalents with known hardware configurations.

Models Selected: The study focuses on the Mistral family (specifically Mistral-7B-Instruct-v0.3 and Mistral-NeMo-Instruct-2407) because they are available in both open-weight (local) and API-accessible versions.
Local Benchmarking:
- Hardware: Experiments were run on six NVIDIA GPUs spanning two generations (Ampere and Hopper) and form factors (SXM and PCIe): A100 (40GB/80GB), A100-PCI, H100, H100-PCI, and H200.
- Workload: Synthetic prompts were generated (covering technical, creative, educational, and business tasks) with two sequence lengths (2,048 and 8,192 tokens).
- Protocol: Each configuration ran 10 independent passes of 100 prompts. Prompts were processed in parallel batches (size 8) to maximize GPU utilization.
- Energy Measurement: Local energy consumption was tracked using CarbonTracker, which samples GPU power via NVIDIA SMI, capturing full board-level energy (including auxiliary components), not just the chip's Thermal Design Power (TDP).
API Benchmarking:
- The same prompts were sent to the Free and Paid versions of the Mistral APIs.
- Runs were scheduled throughout the day to capture load variances.
Estimation Logic:
- The authors assume that at near-optimal utilization, power draw is relatively constant.
- Formula: $E_{api} \approx P_{loc} \times \bar{T}_{api}$ $E_{a p i} \approx P_{l oc} \times \overset{ˉ}{T}_{a p i}$
  - Where $P_{loc}$ is the average power draw measured locally for a specific GPU.
  - $\bar{T}_{api}$ is the mean execution time measured on the API.
- Results were normalized by token count to ensure comparability across different model outputs.

3. Key Contributions

Time-as-Energy Proxy: The paper establishes that inference time per token is a robust proxy for estimating energy consumption in API-based LLMs, provided it is grounded by local measurements.
Hardware Inference: By comparing API inference times against a local "fingerprint" of various GPU generations (Ampere vs. Hopper), the authors demonstrate the ability to infer the underlying GPU architecture used by API providers.
Methodological Framework: The study provides a reproducible protocol for benchmarking API energy costs, highlighting the necessity of normalizing by token count and using local baselines to account for hardware opacity.
Empirical Validation: The work validates that CarbonTracker measurements (real-world power) often exceed TDP estimates, particularly for PCIe cards, emphasizing the need for empirical grounding rather than theoretical calculations.

4. Key Results

Performance Discrepancy: API-based models (both Free and Paid) were significantly faster than any local GPU configuration, achieving roughly 43–54% reduction in computation time compared to the best local setup (H200).
- Interpretation: This suggests API providers use highly optimized, likely multi-GPU, or specialized server-grade hardware (SXM modules) with superior parallelization.
Hardware Identification:
- Mistral-NeMo: The API's time-per-token ( $\bar{T}_{token}$ ) closely matched the H100-PCI local cluster (Hopper generation).
- Mistral-7B: The API times were slightly higher than the H100 cluster but still faster than Ampere (A100) clusters, suggesting a mix of Hopper GPUs or highly optimized server configurations.
Energy Estimation:
- Using the inferred hardware (H100-PCI) and local power data, the authors estimated the energy cost for API runs.
- Mistral-7B: Estimated at ~62 Wh (calculated via TDP) to ~100 Wh (measured via CarbonTracker) for the benchmark.
- Mistral-NeMo: Estimated at ~123 Wh (TDP) to ~203 Wh (CarbonTracker).
- Note: TDP-based calculations consistently underestimated energy use compared to CarbonTracker measurements, with differences ranging from 16% to 43% depending on the GPU architecture.
Free vs. Paid API: The performance difference between Free and Paid API tiers was marginal (within one standard deviation), suggesting they likely share similar backend resources, with variations driven by server load or queueing rather than distinct hardware tiers.

5. Significance and Implications

Transparency for End-Users: This work provides a practical tool for researchers and end-users to estimate the "carbon footprint" of using closed-source LLMs, even when providers do not disclose energy data.
Sustainability Awareness: By quantifying the energy cost of API interactions, the paper highlights the environmental impact of generative AI adoption, which is projected to reach petawatt-hour levels by 2026.
Limitations & Future Work: The authors acknowledge that their model assumes single-GPU dominance. Real-world API deployments may use complex parallelism (tensor/pipeline) which trades latency for higher power consumption. Future work requires ablation studies on these parallelism strategies to refine the proxy accuracy.

Conclusion: The paper successfully demonstrates that inference time is a viable proxy for energy estimation in opaque API environments. By grounding these time measurements against local hardware benchmarks, it is possible to infer the likely GPU infrastructure used by providers and calculate reasonable energy cost estimates, thereby shedding light on the hidden environmental costs of LLMs.