⚡ electrical engineering

Provisioning to Runtime Optimization of a +100 MW AI Cluster

This paper presents the first end-to-end account of power management for a hyper-scale AI datacenter, detailing the process from early power planning for next-generation accelerators to runtime optimization of a 150 MW facility hosting 83,000 GB200 GPUs.

Original authors: Ehsan K. Ardestani, Leonardo Piga, Jovan Stojkovic, Pavan Balaji, Mustafa Ozdal, Mikel Jimenez Fernandez, Mihaela Dimovska, Luka Tadic, Hao Shen, Devika Vishwanath, Richa Mishra, Melaku Mihret, Valent

Published 2026-05-26

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Ehsan K. Ardestani, Leonardo Piga, Jovan Stojkovic, Pavan Balaji, Mustafa Ozdal, Mikel Jimenez Fernandez, Mihaela Dimovska, Luka Tadic, Hao Shen, Devika Vishwanath, Richa Mishra, Melaku Mihret, Valentin Andrei, Mauricio Cespedes, Julien Prigent, James Monahan, Tyler Graf, Bin Li, Charles Marquez, Shobhit Kanaujia, Kaushik Veeraraghavan, Chunqiang Tang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build the world's largest, most powerful supercomputer to solve the hardest problems in Artificial Intelligence. You have bought thousands of the fastest computer chips (GPUs) money can buy. But there's a catch: you don't have enough electricity to run them all at full speed.

In fact, the paper argues that running out of power is now a bigger problem than running out of computer chips. It's like having a fleet of 100 race cars but only enough gas to fill half of them.

This paper describes how Meta (the company behind Facebook) solved this problem for a massive 150-megawatt data center. They didn't just plug everything in and hope for the best; they treated electricity like a precious, limited resource that needed to be managed in three distinct stages.

Here is the story of how they did it, using simple analogies.

The Three Stages of Power Management

The authors break the process down into three phases: Planning, Checking, and Running.

Phase 1: Planning (The "Budgeting" Phase)

Before the computers even arrived, the team had to figure out how many they could fit into their fixed electricity budget.

The Old Way: Usually, engineers look at a chip's "maximum speed" (its Thermal Design Power, or TDP) and assume that's what it will always use. If a chip is rated for 1,200 Watts, they plan for 1,200 Watts.
The Meta Way: They realized that if they forced every chip to run at its absolute maximum, they could only fit a few chips in the building. Instead, they asked: "What if we run all the chips slightly slower?"
The Analogy: Imagine a buffet with a fixed amount of food.
- Strategy A: Give everyone a massive, 10-course meal. You can only feed 50 people.
- Strategy B: Give everyone a slightly smaller, 8-course meal. Now you can feed 60 people.
- The Result: Even though each person got slightly less food, the total amount of food eaten by the group is higher, and more people are fed.
The Discovery: They found that by lowering the power limit of each GPU from 1,200 Watts to about 1,000 Watts, they could fit many more GPUs into the data center. The slight drop in speed for each individual chip was more than made up for by having more chips working together. This gave them a 10% boost in total computing power.

Phase 2: Checking (The "Reality Check" Phase)

Once the computers were installed, the team realized their plans didn't match reality perfectly.

The Problem: The computers' internal power meters (PSUs) were lying. They were being overly cautious, reporting that the machines were using more power than they actually were. It's like a car's fuel gauge that always says "Empty" when you still have a quarter tank left.
The Fix: They compared the internal meters to the building's main power sensors (which are very accurate). They found that if they ignored the "worst-case" readings and looked at the "average" usage (specifically the 70th percentile), they could trust the numbers more.
The Result: Because they stopped trusting the overly cautious internal meters, they realized they had a little bit of "hidden" power they weren't using. They turned the power back up slightly (from 1,000W to 1,020W), squeezing out another 2% performance boost.

Phase 3: Running (The "Traffic Cop" Phase)

Now the data center is running live AI training jobs. The problem is that AI workloads are "synchronous," meaning all the chips have to wait for each other. If one chip is slow, the whole job slows down.

The Problem: Sometimes, the power grid gets a sudden spike. If the system detects a spike, the old way was to immediately shut down or slow down one specific chip to save power. But in a synchronized race, if you slow down one runner, the whole team loses.
The Solution (Dimmer): They built a smart system called "Dimmer." Instead of punishing one chip, it gently lowers the power of every chip in the job by a tiny, equal amount.
The Analogy: Imagine a group of runners in a relay race. If the track gets slippery (power limit), instead of tripping one runner (which stops the whole team), the coach tells everyone to run just a tiny bit slower. The team stays together, and the race continues without stopping.
The Result: This prevents the system from crashing and allows them to use the last few percent of "stranded" power that was previously wasted. This added another ~2% boost.

Other Key Insights

The "Bottleneck" Effect: The data center is a mix of different hardware (computers, cooling, network switches). Sometimes, the power limit isn't set by the computers, but by the cooling system or the network cables. If one part of the chain is weak, it limits the whole chain.
Power Swings: When AI chips work, they sometimes pause for a split second to talk to each other, causing the power usage to dip and then spike. The team created a "software smoother" that fills these tiny gaps with dummy work, keeping the power usage steady like a calm river instead of a choppy sea. This protects the building's electrical grid from shocks.
Newer is Better: They compared their new chips (GB200) to older ones (H100). The new chips are so efficient that even though they use less power per chip, the new cluster is nearly twice as fast as the old one would have been in the same building.

The Bottom Line

By treating electricity as a flexible resource rather than a fixed limit, and by managing it carefully from the planning stage to the daily operations, Meta managed to get roughly 14% more computing power out of their 150-megawatt data center without building a single new wall or buying a single new wire.

They didn't find a magic battery; they just learned how to use the electricity they already had much more wisely.

Technical Summary: Provisioning to Runtime Optimization of a +100 MW AI Cluster

Problem Statement

The primary bottleneck in the race toward Artificial General Intelligence (AGI) has shifted from the availability of AI accelerators to the electric power supply for datacenters. While previous research has focused on lowering utility costs, optimizing green energy usage, or exploiting power oversubscription, these studies often overlook the critical reality that datacenter power planning must prioritize maximizing total compute capacity within a fixed power budget. Furthermore, prior work typically addresses only the final operational phase of power management, neglecting the end-to-end process that spans from early planning (6–12 months before hardware availability) to deployment validation and dynamic runtime tuning.

This paper addresses the challenge of managing power for a hyperscale AI cluster (150 MW, 83,000 GB200 GPUs) by presenting a holistic, three-phase approach: Provisioning, Deployment Validation, and Operational Runtime Optimization.

Methodology

The authors propose a three-phase framework for power management, utilizing a 150 MW datacenter hosting 83,000 GB200 GPUs connected via an RDMA backend network.

Phase 1: Planning and Provisioning

In this phase, the goal is to determine the optimal cluster configuration to maximize aggregate throughput ( $T(p)$ ) within a fixed power budget ( $P_{total}$ ).

Optimization Model: The authors formulate the problem as maximizing $T(p) = N(p) \cdot f(p)$ , where $N(p)$ is the number of GPUs that fit the budget and $f(p)$ is the per-GPU performance at power limit $p$ .
Power Limit Selection: Instead of using the Thermal Design Power (TDP) of 1200W, the team analyzes the performance-per-watt curve. They find that reducing the power limit allows for higher GPU density. Empirical data shows that while HBM bandwidth drops sharply below 1000W, it remains constant between 1000W and 1200W. Consequently, they select a 960W (80% of TDP) power limit as the optimal operating point, balancing per-GPU performance loss against the gain in total GPU count.
Holistic Provisioning: The methodology accounts for the entire power delivery hierarchy (Main Switchboards, Switchboards, Reactor Power Panels) and includes non-GPU power consumption (cooling, networking, CPUs).

Phase 2: Deployment Validation

Once hardware is deployed, the team validates assumptions and refines power settings based on real-world telemetry.

Measurement Correction: The authors identify that Power Supply Unit (PSU) readings systematically overestimate actual power consumption. To correct this, they compare PSU data against high-accuracy oscilloscope and Reactor Power Panel (RPP) DCIM sensors. They determine that using the 70th percentile (P70) of PSU samples within a minute provides the most accurate estimate of aggregate rack power, minimizing the bias of maximum PSU values.
Headroom Analysis: The study reveals significant static power imbalance across the datacenter due to heterogeneous hardware mixes (GPU racks, network racks, cooling) and physical placement constraints. This imbalance leaves approximately 5–10% of power "stranded" at the most constrained Main Switchboards (MSBs).
Adjustment: Based on validation data, the team increases the operational power limit from the planned 960W to 1020W, ensuring the P70 power consumption aligns with the rack budget.

Phase 3: Active Runtime Optimization

This phase focuses on dynamic power management to capture remaining stranded power and handle transient spikes without disrupting training jobs.

Power Swings: Synchronous training algorithms cause synchronized power oscillations. To mitigate this, the authors implement a "gap filling" technique using a software-based power smoother. This kernel generates synthetic load on Tensor Cores during communication gaps, maintaining a stable power draw (up to 800W per accelerator) with less than 3% performance overhead.
Dimmer Algorithm: To manage power capping dynamically, they developed "Dimmer." Unlike traditional capping that forces abrupt drops or task eviction, Dimmer gradually reduces power limits across all GPUs in a job when a power device approaches its limit (e.g., 97%).
- Scheduler Awareness: Dimmer integrates with the job scheduler to prioritize large, mission-critical pre-training jobs. It distributes power reductions evenly to avoid creating "stragglers" that would stall the entire job.
- Safety Mechanisms: The system uses a 7-second averaging window to distinguish between transient spikes and sustained overages, aligning with circuit breaker trip characteristics. A fail-safe mechanism ensures hosts revert to a safe global limit if communication with the controller is lost.

Key Contributions and Findings

End-to-End Power Management: This is the first paper to describe the complete power management lifecycle for a hyperscale AI cluster, from pre-deployment planning to runtime optimization.
Optimal Power Limiting: The study demonstrates that operating GPUs below their maximum TDP (specifically at 960W initially, later validated to 1020W) maximizes total cluster throughput. Reducing power from 1200W to 1000W saves 16.7% power with only a 5% performance drop, allowing for significantly higher GPU density.
Measurement Accuracy: The paper identifies that PSU telemetry overestimates power usage. Adopting a P70 aggregation method corrects this bias, enabling more precise power budgeting.
Static Imbalance: Hardware heterogeneity and physical constraints create uneven power headroom across the facility. The most constrained MSBs dictate the safe operating range, leaving up to 10% of power stranded if not managed dynamically.
Dynamic Optimization: The "Dimmer" system and power smoothing techniques successfully capture the final percentage of stranded power. By avoiding abrupt throttling and distributing load reductions, the system maintains job stability while maximizing utilization.

Results

The paper quantifies the throughput improvements achieved through each phase of their methodology, comparing the final optimized state against a baseline of 1200W per GPU:

Provisioning Phase: Optimizing for performance-per-watt (960W limit) increased cluster throughput by 10% compared to the 1200W baseline by enabling higher GPU density.
Deployment Validation: Refining the power limit to 1020W based on real-world telemetry provided an additional ~2% throughput increase.
Runtime Optimization (Dimmer): Dynamic management of stranded power and power swings yielded a further ~2% improvement.
Total Gain: The cumulative effect of these strategies resulted in a ~14% total improvement in cluster throughput compared to the initial 1200W configuration.

Additionally, the paper notes that compared to the H100 generation, the GB200 at 960W delivers 2.4x the per-GPU performance and 1.9x the aggregate cluster throughput within the same power budget.

Significance and Claims

The authors position this work as a foundational guide for the industry, moving beyond cost-centric power management to capacity-centric strategies essential for AGI development. They claim this is the first production-scale report covering all three phases of power management for a 150 MW AI cluster.

The paper emphasizes that power supply is now the primary limiting factor for AGI, surpassing accelerator availability. By sharing insights on power provisioning, measurement correction, and dynamic runtime tuning, the authors aim to encourage other practitioners to share their experiences and collectively advance the efficiency of AI infrastructure. They explicitly state that their goal is not to propose new theoretical algorithms but to share practical, production-tested experiences and findings that can guide future practitioners in designing efficient, scalable AI datacenters.