Commissioning and Thermohydraulic Characterization of a Single-Phase Liquid-Cooled High-Density Data Center Rack

Abstract High-density data centers are currently increasing in demand. As data centers grow in power and complexity, traditional air cooling is no longer adequate for handling contemporarily increasing HPC and AI workloads. Liquid cooling is emerging as an efficient solution for high-density data center cooling, with a host of benefits, including reduced costs, higher sustainability, and the ability to handle higher power densities.

In the current work, a single-phase liquid-cooled test high-density rack was assembled, experimentally commissioned, and characterized at full design power. The rack has four servers, each with eight (8) high-power graphic processing units (GPU) thermal test vehicles (TTV) and four (4) low-power switch TTVs. Each TTV was instrumented with Type-T thermocouples to measure the case temperatures.

Each GPU TTV was powered at 1000 W and each switch TTV was powered at 250 kW. Liquid-cooled cooling loops (each consisting of eight (8) GPU cold plates and four (4) switch cold plates) were attached to each of the server TTVs. The inlet and outlet of each cooling loop were instrumented with calibrated thermistors, pressure sensors, and magnetic-inductive flow meters to measure the cooling loop’s inlet and exit fluid temperatures, pressure drop, and flow rates, respectively.

Two of the test servers were reliability servers and the other two were the thermal verification server and hydraulic verification server. The thermal verification server was hugely instrumented with numerous thermistors to verify the inlet and outlet fluid temperatures of each cold plate. The hydraulic verification server had multiple magnetic-inductive flow meters to verify flow distribution in each cold plate on the cooling loop.

Results show that at the coolant [25% propylene glycol/water (PG-25)] supply temperature of 32 °C and full design power, the average GPU TTV case temperature per TTV was close to 64.5 °C. The mean switch heater Tcase is generally identical per server TTV and is about 53.5 °C, with a mean standard deviation of less than 0.7 °C. The energy balance on a rack reveals that about 97.1% of the heat dissipated by server TTVs was rejected to primary side/facilities chilled water.