Digital Twin-Based Cooling System Optimization for Data Center

This paper presents a validated digital twin of the Frontier supercomputer's liquid cooling system to demonstrate that a ramp-constrained, joint optimization of flow rate and supply temperature can achieve 27.8% energy savings, significantly outperforming flow-only strategies by addressing the gap between theoretical optima and operational deployability.

The Gist

Imagine a massive, high-performance supercomputer named Frontier. It's so powerful it can solve problems in seconds that would take a regular computer millions of years. But there's a catch: Frontier gets incredibly hot, like a car engine running at full speed in the middle of a desert. To keep it from melting, it needs a constant, powerful cooling system.

This paper is about how two engineers figured out how to make that cooling system smarter, safer, and much cheaper to run.

Here is the story of their discovery, explained simply:

1. The Problem: The "Over-Protective" Cooling System

Right now, Frontier's cooling system works a bit like a nervous parent driving a car.

• The Reality: The computer generates heat, and the cooling system pumps cold water through pipes to absorb that heat.
• The Issue: The current system is too cautious. It pumps water at a high, constant speed, regardless of whether the computer is actually working hard or just taking a nap. It's like driving a car at 80 mph even when you're stuck in a parking lot.
• The Cost: This "over-pumping" wastes a massive amount of electricity. In fact, the fans that blow air to cool the water (the cooling towers) and the pumps moving the water are eating up a huge chunk of the data center's power bill.

2. The Solution: Building a "Digital Twin"

Before they could try to fix the real machine, the engineers built a Digital Twin.

• The Analogy: Imagine you have a real, expensive race car. You don't want to crash it to see how fast it can go. So, you build a perfect, virtual copy of it inside a computer. You can crash the virtual car a thousand times, change the tires, and tweak the engine without spending a dime or breaking anything.
• What they did: They created a virtual version of Frontier's cooling system using real data from a whole year. They tested their ideas on this "ghost machine" first to make sure they wouldn't accidentally overheat the real computer.

3. The Three Strategies: From "Good" to "Great"

The engineers tested three different ways to run the cooling system, like trying three different driving styles:

• Strategy A: The "Flow-Only" Driver (The Basic Fix)

  • The Idea: Just slow down the water pump when the computer doesn't need as much cooling.
  • The Result: This saved about 20% of the energy. It was a good start, like taking your foot off the gas pedal when you hit a red light. But it wasn't the best possible outcome.

• Strategy B: The "Unconstrained" Driver (The Theoretical Perfect)

  • The Idea: What if we could change everything instantly? We could slow down the pump and simultaneously raise the temperature of the water entering the system (making the cooling fans work less hard).
  • The Catch: In the real world, you can't change settings instantly. If you change the water flow too fast, you get "water hammer" (a loud, damaging shockwave in the pipes). If you change the temperature too fast, you might crack the metal equipment.
  • The Result: This "perfect" strategy saved 30% of the energy. It was the theoretical maximum, but it was too risky to actually use.

• Strategy C: The "Smart & Safe" Driver (The Real-World Winner)

  • The Idea: This is the sweet spot. It uses the smart logic of Strategy B but adds "guardrails." It says, "You can change the settings, but only slowly and smoothly, just like a human driver would."
  • The Result: This saved 27.8% of the energy. It captured 92% of the "perfect" savings while being safe enough to install on the real machine.

4. The Big Surprise: It's Not Just About the Pump

The most interesting discovery was a counter-intuitive finding.

• The Old Way: Everyone thought the biggest waste was the water pump, so they tried to slow it down as much as possible.
• The New Discovery: The engineers found that the cooling tower fans (the big fans blowing air) actually use 73% of the total energy, not the pumps!
• The Metaphor: Imagine you are trying to cool a room. You thought the problem was the fan blowing the air, so you turned it down. But the real problem was that the air coming into the fan was too cold. By letting the water get slightly warmer (which is safe for the computer), the cooling fans didn't have to work as hard.
• The Lesson: Sometimes, to save energy, you have to let the system run "hotter" (but still safely) so the other parts don't have to work so hard.

5. The "Implementability Gap": Theory vs. Reality

The paper introduces a new concept called the "Implementability Gap."

• The Concept: This is the difference between what is mathematically perfect and what is practically possible.
• The Finding: Usually, when you add real-world rules (like "don't change settings too fast"), you lose a lot of your potential savings. But here, the gap was tiny. The "Smart & Safe" strategy (Strategy C) got almost all the benefits of the "Perfect" strategy. This means we don't have to choose between being safe and being efficient; we can be both.

Summary

This paper is a victory for smart engineering. By building a virtual copy of a supercomputer's cooling system, the researchers proved that:

1. We are currently wasting a lot of money by being too cautious.
2. We can save nearly 30% of the energy used for cooling.
3. We can do this safely by making small, smooth adjustments rather than drastic changes.

It's like realizing that your car gets better gas mileage if you drive smoothly and let the engine warm up a bit, rather than constantly slamming on the brakes and accelerating hard. The same logic applies to the world's most powerful computers.

Technical Summary

1. Problem Statement

Data centers, particularly High-Performance Computing (HPC) facilities like the Frontier exascale supercomputer, consume massive amounts of electricity, with cooling systems accounting for 30–40% of total consumption. While optimization studies exist, they often fail to address the "implementability gap": the discrepancy between theoretically optimal energy savings and what is practically achievable under real-world operational constraints (e.g., actuator ramp-rate limits, thermal inertia, and safety margins).

Existing approaches often rely on data-driven methods (like Reinforcement Learning) that lack interpretability or physics-based models that are not coupled with systematic optimization. Furthermore, no prior study has formally quantified how much theoretical energy savings are lost when practical constraints are enforced in exascale liquid cooling systems.

2. Methodology

2.1 System Description

The study focuses on the Frontier supercomputer at Oak Ridge National Laboratory (ORNL), which uses a warm-water liquid cooling architecture. The system consists of:

• Three parallel subloops (active) within the Hot-Temperature Water (HTW) loop.
• Coolant Distribution Units (CDUs) circulating ethylene glycol-water mixtures through compute cabinets.
• Plate heat exchangers transferring heat from the HTW loop to a primary Cooling Tower Water (CTW) loop.
• Mechanical-draft cooling towers rejecting heat to the atmosphere.
• Variable-speed pumps driving the HTW flow.

The dataset covers one full calendar year (2023) with 10-minute intervals, comprising 47,186 validated operational records.

2.2 Digital Twin Development

The authors developed a physics-based digital twin using the Modelica language and the Buildings Library within the OpenModelica environment.

• Model Components:
  • Heat Exchangers: Modeled using the effectiveness-NTU ($\varepsilon$-NTU) method.
  • Pumps: Modeled using affinity laws ($P \propto \dot{m}^n$), where $n$ is tested between 2.0 and 3.0.
  • Cooling Towers: Modeled using a fixed-approach temperature formulation.
• Calibration & Validation: The model was calibrated against operational data, specifically adjusting the effective glycol mass fraction to match specific heat capacity ($c_p$) values. Validation followed ASHRAE Guideline 14, achieving a Coefficient of Variation of Root Mean Square Error (CV-RMSE) of 1.96–2.67% and a Normalized Mean Bias Error (NMBE) within ±2.5% across all subloops.

2.3 Layered Optimization Framework

The study introduces a three-tiered optimization strategy to systematically evaluate energy savings and the implementability gap:

1. Strategy A (Flow-Only): Optimizes pump flow rate only, keeping supply temperature fixed at baseline. This establishes a conservative baseline.
2. Strategy B (Unconstrained Co-Optimization): Co-optimizes pump flow rate and supply temperature setpoint without operational constraints. This reveals the theoretical maximum savings.
3. Strategy C (Ramp-Constrained Co-Optimization): Re-introduces practical constraints, specifically ramp-rate limits on flow ($\pm 50$ kg/s per 10-min step) and supply temperature ($\pm 1^\circ$C per 10-min step). This yields the practically implementable solution.

Key Metric: The Implementability Gap is defined as the energy savings lost when moving from Strategy B to Strategy C. The Recovery Ratio measures the percentage of theoretical savings retained under constraints.

3. Key Contributions

1. Validated Digital Twin: Creation of a high-fidelity Modelica-based digital twin for an exascale liquid cooling system, calibrated against a full year of real-world data.
2. Layered Optimization Framework: The first application of classical gradient-based optimization (SLSQP) to HPC cooling that explicitly separates theoretical potential from practical feasibility.
3. Implementability Gap Metric: A formal definition and quantification of the energy penalty incurred by operational constraints, providing a new standard for evaluating control strategies.
4. System-Level Insights: Demonstration that optimizing pump flow alone is suboptimal; co-optimizing supply temperature is critical due to the dominance of cooling tower fan energy.

4. Key Results

4.1 Baseline Inefficiency

• The baseline system operates at 2.9 times the minimum thermally safe flow rate.
• There is systematic over-pumping, with a median over-pumping ratio of 1.5x, leading to a cubic increase in pump power waste.
• Cooling Tower (CT) Fan Energy dominates the budget, accounting for 73% of total cooling energy consumption.

4.2 Energy Savings Comparison

Strategy	Approach	Total Energy Savings	Pump Savings	CT Fan Savings
Baseline	Fixed-speed setpoints	—	—	—
Strategy A	Flow-only optimization	20.4%	75.7%	0.0%
Strategy B	Unconstrained Co-Optimization	30.1%	63.1%	17.9%
Strategy C	Ramp-Constrained Co-Optimization	27.8%	60.5%	15.8%

• Strategy B vs. A: Co-optimizing supply temperature (raising it by an average of $+2.7^\circ$C) nearly doubles the savings compared to flow reduction alone. This is because raising the supply temperature reduces the heat rejection burden on the cooling towers, which is the larger energy consumer.
• Strategy C vs. B: Enforcing ramp-rate constraints results in a 2.3% absolute reduction in total savings (from 30.1% to 27.8%).
• Recovery Ratio: Strategy C recovers 92.4% of the theoretical maximum savings, proving that near-optimal performance is achievable without violating actuator limits.

4.3 Robustness

The results are robust to uncertainty in the pump power exponent ($n$). Even under a conservative assumption ($P \propto \dot{m}^2$), Strategy C achieves 21.3% total savings.

5. Significance

• Paradigm Shift: The paper challenges the conventional focus on pump flow reduction alone. It demonstrates that in systems where cooling towers dominate energy use, raising the supply temperature (within safe limits) to reduce fan load is more effective than minimizing pump power.
• Practical Feasibility: By quantifying the "implementability gap," the authors show that theoretical optimization is not just a theoretical exercise; 92.4% of potential savings can be realized in a real-world deployment with smooth, safe control trajectories.
• Scalability: The methodology is applicable to other liquid-cooled data centers and HPC facilities. The use of a physics-based digital twin ensures generalizability and interpretability, avoiding the "black box" issues of purely data-driven AI approaches.
• Policy Implication: As HPC facilities scale toward multi-exascale power envelopes, systematic optimization of cooling infrastructure is critical for meeting global energy efficiency targets. This work provides a blueprint for quantifying and bridging the gap between theoretical efficiency and operational reality.