Co-Design Optimization for Data Center Cooling System via Digital Twin

Technical Summary: Co-Design Optimization for Data Center Cooling System via Digital Twin

Problem Statement
Liquid-cooled exascale supercomputers, such as the Frontier system at Oak Ridge National Laboratory, dissipate massive amounts of heat (8–30 MW) through cooling plants organized into multiple parallel subloops. While operational data collection and control strategies have been studied, a systematic approach to the design of these facilities remains underdeveloped. Specifically, the optimal allocation of Coolant Distribution Units (CDUs) across parallel subloops and the dynamic distribution of coolant flow among them have not been jointly optimized. Existing literature often treats the plant topology as fixed or focuses solely on operational control without addressing the combinatorial design space of CDU partitioning and flow allocation. This paper addresses the gap in co-design optimization, seeking to determine the optimal integer partition of CDUs, continuous flow fractions, and per-timestep operational setpoints to minimize annual cooling energy consumption.

Methodology
The authors propose a three-layer optimization framework applied to a reduced-order physics-based digital twin of the Frontier cooling system, calibrated against the Stage 1 Modelica model and validated using 2023 operational data (49,353 timesteps).

1. Layer 1: Topology Design (Integer Partition): This layer determines the integer partition of $N=25$ CDUs into $K$ parallel subloops. The design space includes all feasible partitions for $K \in \{2, 3, 4, 5, 6\}$, totaling 611 distinct configurations. The framework evaluates both "balanced" (CDUs sorted by load) and "worst-case" (high-load CDUs grouped together) assignment policies.
2. Layer 2: Flow Fraction Optimization (Continuous Allocation): This layer determines the continuous flow fractions ($f_k$) allocated to each subloop. These fractions can be fixed (proportional to CDU count) or dynamically optimized at each timestep to balance thermal intensity across subloops.
3. Layer 3: Operational Co-Design (Control): This layer optimizes the total mass flow rate ($\dot{m}$) and supply temperature setpoint ($T_{sup}$) at each timestep. The objective is to minimize total plant power (pump + cooling tower fan) subject to per-subloop thermal safety constraints (return temperature $\le 42^\circ$C).

The framework solves the mixed-integer nonlinear problem by exhaustively enumerating all 611 integer partitions (Layer 1) and solving the continuous subproblem (Layers 2 and 3) using Sequential Least-Squares Programming (SLSQP) with analytical gradients. Three operational strategies are compared:

• Strategy A: Flow-only optimization (fixed temperature, fixed proportional flow).
• Strategy B: Co-design of flow and temperature with fixed proportional flow.
• Strategy C: Full three-layer optimization with dynamically adjusted flow fractions.

Key Contributions

1. Three-Layer Framework: Development of a decomposition approach that separates integer topology decisions from continuous operational decisions, providing a global optimality certificate over the discrete design space via exhaustive enumeration.
2. Comprehensive Evaluation: Application of the framework to the Frontier system, evaluating all 611 feasible CDU partitions across a full year of operational data under both balanced and worst-case assignment scenarios.
3. Design Hierarchy: Establishment of a decision hierarchy demonstrating that the choice of operational strategy dominates topology decisions, which in turn dominate CDU-to-subloop allocation.
4. Flow Fraction Compensation: Demonstration that dynamic flow fraction optimization can compensate for suboptimal CDU assignments, effectively decoupling the design-layer partition choice from the final energy outcome.

Results

• Optimal Topology: The globally optimal design is a two-subloop plant with a CDU partition of (19, 6), achieving 35.48% annual cooling energy savings. The current Frontier design (three subloops, partition 14-6-5) achieves 35.30% savings under the same optimized strategy, placing it only 0.18% below the global optimum.
• Strategy Impact: Moving from flow-only optimization (Strategy A, ~23.4% savings) to full three-layer co-design (Strategy C) yields a 12.0% improvement in energy savings.
• Flow Optimization Value: Optimizing flow fractions (Strategy C vs. Strategy B) provides an additional 0.2–0.5% savings. Crucially, flow optimization reduces the sensitivity of energy performance to the specific CDU-to-subloop assignment by 93%. Under fixed flow fractions, the difference between best and worst partitions is ~5.3%; with optimized flow, this gap collapses to ~0.4%.
• Workload Equalization: The study finds that dynamic flow redistribution acts as a functional equivalent to workload equalization. While equalizing the computational load across CDUs passively reduces energy use, active flow control achieves the same thermal intensity equalization without requiring changes to the HPC job scheduler.

Significance and Claims
The paper claims that for the Frontier facility, the most practical and cost-effective improvement is not a physical reconfiguration of the cooling plant topology (e.g., converting from three to two subloops), but rather the activation of independent per-subloop flow control via a software update. This "software-only" retrofit can recover approximately 10 MWh/yr of additional energy savings relative to a fixed proportional baseline, with negligible capital cost.

The authors assert that the proposed framework is transferable to other liquid-cooled high-performance computing (HPC) plants. They position this work as the third stage in a progression of cooling optimization: moving from data-driven waste quantification to validated physics-based control, and finally to joint design and operational co-design. The methodology offers a computationally tractable alternative to general mixed-integer nonlinear programming for systems where the integer design space is small enough for exhaustive enumeration, providing a provable global optimum that branch-and-bound methods cannot guarantee for nonconvex thermal-hydraulic models.