⚡ electrical engineering

The Hidden Energy Cost of Scaling AI Hardware: A Comparative Analysis of Manufacturing and Operational Energy

This paper reveals that the rapidly growing energy embodied in manufacturing AI hardware, particularly high-bandwidth memory, is a critical and often overlooked environmental cost that will soon outweigh operational energy consumption, necessitating a broader framework for hardware sustainability assessments.

Original authors: Emre Salman, Abrar Abdurrob

Published 2026-07-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Emre Salman, Abrar Abdurrob

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world of artificial intelligence as a massive, high-speed train race. For years, everyone has been obsessed with how fast the train moves while it's running—the speed of the engine, the efficiency of the fuel, and how many passengers (data) it can carry per gallon of electricity. This is what scientists call "operational energy." It's the power you pay for when you flip a switch and the computer starts working. But there's a hidden part of the journey that often gets ignored: the cost of building the train itself. Before a single passenger boards, thousands of tons of steel, glass, and copper must be mined, melted, and assembled. In the world of computer chips, this is the "manufacturing energy." It's the electricity used to carve the tiny circuits into silicon and stack the memory layers that hold the AI's brain. The question isn't just about how much energy the AI uses while thinking; it's about how much energy it took to build the thinker in the first place.

This paper, written by researchers Emre Salman and Abrar Abdurrob, decides to look at the whole picture, not just the running costs. They zoom in on the specific hardware that powers the smartest AI models today: the datacenter accelerators. These are the super-chips that train giant language models. The authors discovered a surprising imbalance. While we have gotten really good at making these chips run efficiently (using less electricity per task), the cost to build them is skyrocketing. The main culprit? A special type of super-fast memory called High-Bandwidth Memory (HBM). Think of HBM as a massive library of books stacked right next to the brain so the brain can grab information instantly. To keep up with AI's hunger for data, manufacturers are stacking dozens of these memory layers on top of each other. The paper finds that this stacking process is so energy-intensive that it now accounts for more than half of all the energy used to create the chip, even before the chip is turned on.

The researchers compared the "birth cost" (manufacturing) with the "life cost" (operation) of these chips. They found that for a single AI accelerator, the energy used to build it is roughly 146 kilowatt-hours (kWh) today. However, the energy it uses while running for three years is much higher, ranging from about 6.4 to 12.8 megawatt-hours (MWh)—that's thousands of times more than the building cost. So, for now, the running cost still wins. But here is the twist: the paper suggests that while we are getting better at making the chips run efficiently, the cost to build them is about to explode. By 2035, the energy required to manufacture a single AI chip could jump six to nine times higher than it is today.

Why is this happening? The authors point out that the latest tricks to make AI smarter and faster—like using "sparse" models that only wake up a few parts of the brain at a time—don't actually shrink the memory library. Even if the AI only uses a tiny fraction of its knowledge at any given moment, it still needs to have all that knowledge stored in the HBM stacks to be ready instantly. So, while the chip might use less electricity to run a specific task, the factory still has to build a massive, energy-hungry memory stack to hold all the unused data. The paper argues that we are stuck in a trap: every time we make the AI more capable by adding more memory, the "embodied" energy cost of building that memory goes up, regardless of how efficiently the chip runs later.

The study also looks at the future. If current trends continue, the manufacturing energy for these chips could reach between 1,141 and 1,660 kWh per device by 2035. This is a six- to nine-fold increase from today. The authors suggest that simply making the chips run better won't fix this problem because the two types of energy are driven by different things. Making the chip run faster is like tuning the engine; making the chip cheaper to build would require changing the engine's design entirely, perhaps by using different materials or stacking methods that don't require so much energy to create.

In short, the paper warns that if we only look at how much electricity an AI datacenter uses while it's working, we are missing a huge part of the environmental story. The "hidden cost" of building the hardware is growing fast, and the clever software tricks we use to save power today might not be enough to stop the total energy bill from climbing. The authors conclude that to truly understand the environmental impact of AI, we need to start counting the energy it takes to build the machines, not just the energy it takes to run them. They suggest that future solutions might need to focus on reducing the sheer amount of memory we need to manufacture, rather than just trying to make the existing memory run more efficiently.

Technical Summary: The Hidden Energy Cost of Scaling AI Hardware

Problem Statement
Current assessments of the environmental footprint of Artificial Intelligence (AI) infrastructure focus almost exclusively on operational electricity consumption (training and inference). This framing omits a critical and growing component: the embodied energy required to manufacture the hardware itself. As AI models scale, the industry relies on hardware architectures that prioritize memory capacity and bandwidth, specifically through High-Bandwidth Memory (HBM) stacks. The authors argue that while operational efficiency per token is improving through architectural innovations, these gains do not translate to reductions in manufacturing energy. Instead, the manufacturing energy per accelerator is projected to rise significantly, creating a divergence between the two phases of the hardware lifecycle that is currently unaccounted for in sustainability metrics.

Methodology
The study employs a lifecycle energy analysis framework to quantify both manufacturing and operational energy contributions across current and projected AI accelerator generations (through 2035).

Manufacturing Energy Estimation: The authors utilize a wafer-level scaling and per-die allocation framework. This approach calculates total manufacturing energy by summing contributions from logic fabrication, memory fabrication, and advanced packaging.
- Wafer Energy: Derived from foundry data (e.g., TSMC) and scaled by process node complexity using the Power-Performance-Area-Cost-Environmental (PPACE) scaling factors. The model accounts for the increased energy intensity of Extreme Ultraviolet (EUV) lithography at advanced nodes (3 nm, 2 nm, and sub-2 nm).
- Die Allocation: Energy is allocated to individual dies based on die area and yield models (negative-binomial yield), acknowledging that larger dies suffer from lower yields, thereby increasing the energy cost per functional die.
- Components: The analysis explicitly separates logic dies, HBM stacks (DRAM and base logic dies), and advanced packaging (e.g., CoWoS interposers).
Operational Energy Estimation: Operational energy is modeled using subsystem-level power decomposition (compute, memory, interconnect) and empirical activity factors.
- Datacenter: Estimates incorporate Thermal Design Power (TDP), activity factors ( $\alpha$ ) for active workloads, utilization rates ( $U$ ), and idle power fractions ( $\beta$ ). Facility-level energy includes Power Usage Effectiveness (PUE) overheads.
- Mobile: Uses duty-cycle models based on active use and connected-standby states.
Projections: Future trends (2024–2035) are grounded in vendor roadmaps (NVIDIA, AMD, Intel, Google), JEDEC standards (HBM4, HBM5), and IRDS lithography roadmaps.

Key Contributions and Results

Dominance of HBM in Manufacturing Energy:
The analysis reveals that HBM fabrication is the primary driver of manufacturing energy in datacenter accelerators. For the NVIDIA H100, HBM accounts for 55% of the total manufacturing energy (~80 kWh out of ~146 kWh). This share increases for devices with larger memory configurations, reaching 71% for the Google TPU v5p. The sheer number of stacked DRAM dies required to sustain memory bandwidth outweighs the energy contribution of the logic die, even when the logic is fabricated at a more advanced node.
Divergence of Manufacturing and Operational Trajectories:
The paper identifies a structural gap between manufacturing and operational energy trends.
- Operational Efficiency: Architectural advances such as Mixture-of-Experts (MoE) sparsity, attention compression (e.g., Multi-head Latent Attention), and precision quantization improve operational energy per token by reducing active compute or memory bandwidth utilization.
- Manufacturing Cost: These innovations do not reduce the total memory capacity required. MoE models still require all expert weights to reside in HBM, and extended context windows require larger KV caches. Consequently, as models scale in total parameter count, HBM capacity (and thus manufacturing energy) continues to grow, unaffected by operational efficiency gains.
Projected Growth Through 2035:
Projecting vendor roadmaps to 2035, the study estimates that manufacturing energy per accelerator will grow six- to nine-fold (from ~176 kWh for an H200 to 1,141–1,660 kWh).
- Logic: Expected to grow ~8–10× due to node scaling (transitioning to sub-1 nm and Å-class nodes) and multi-chiplet integration.
- HBM: Expected to grow ~5–9× driven by increasing stack heights (from 8-high to 16–20-high) and die counts (scaling toward 800 GB–1.5 TB per accelerator).
- Packaging: Expected to grow ~13–16× due to the expansion of multi-reticle interposers.
Comparison with Mobile Platforms:
In contrast to datacenter accelerators, where operational energy exceeds manufacturing energy by two orders of magnitude, mobile SoCs exhibit a near-parity between manufacturing and operational energy over a three-year lifetime (both in the range of 4–13 kWh). This suggests that for mobile devices, extending device longevity is a sustainability lever comparable to improving chip efficiency.
The "Irreversibility Asymmetry":
The authors highlight that manufacturing energy is "locked in" at fabrication and cannot be reduced by subsequent software optimization or operational efficiency. Conversely, operational energy can be reduced over time. This creates an "embodied debt" that compounds with every hardware refresh cycle, a factor invisible to operational-only metrics.

Significance and Claims
The paper claims that current sustainability assessments are incomplete because they fail to account for embodied manufacturing energy, particularly the energy-intensive scaling of HBM. The authors argue that:

Hardware sustainability assessments must explicitly account for embodied manufacturing energy alongside operational consumption.
Architectural strategies targeting memory capacity directly (rather than just bandwidth or compute) may offer lifecycle energy benefits that operational efficiency improvements alone cannot achieve.
The "Jevons paradox" applies to AI hardware: Efficiency gains in operational energy may lead to increased aggregate demand and fleet sizes, further compounding the total lifecycle energy burden due to the rising manufacturing cost per device.
Reporting frameworks need to evolve: Current industry and regulatory reporting (e.g., ISO-aligned disclosures) often defaults to operational metrics. The authors suggest that metrics treating embodied CO2e and total lifecycle energy as primary optimization targets are necessary to capture the true environmental cost of AI scaling.

The study concludes that the decoupling of manufacturing and operational energy is a fundamental characteristic of complex semiconductor systems, driven by independent architectural variables (capacity vs. utilization/bandwidth), and requires distinct interventions to address.

Technical Summary: The Hidden Energy Cost of Scaling AI Hardware

More like this