Energy-Efficient, High-Performance Computing at Pacific Northwest National Laboratory: Spray-cooling Demonstration

Phillip Tuma, P.E., 3M Company
Tahir Cader, SprayCool, Inc.

The U.S. Environmental Protection Agency (EPA) studied the issue of escalating power consumption in U.S. data centers. The EPA report, issued August 4, 2007, found that electricity usage by data centers accounted for 1.5 percent of the total electricity used in the U.S. in 2006, and that, if current data center operating practices continue, electricity usage will almost double to 2.9 percent of the total electricity used in the U.S. in 2011 (EPA, 2007). Additional key conclusions and recommendations from the report include:

• Very few data center operators measure or even know the efficiency of their data centers.
• The federal government should lead by example in improving data center energy efficiency.
• The federal government should collaborate with leading industry organizations that are focused on the issue and represent a consolidated industry response.

In the spirit of responding to the findings and recommendations of the EPA report, the Pacific Northwest National Laboratory (PNNL) has teamed with several key organizations including The Green Grid (TGG), ASHRAE TC9.9, IBM, 3M, and SprayCool. As a U.S. Department of Energy (DOE) National Laboratory, PNNL has been engaged in Energy Smart data center research since 2003. In the latest phase of work, PNNL has teamed with IBM and SprayCool to install a spray cooled 9.3 TFlop IBM x3550 cluster (#186 on the TOP500 2008 Fall list). The cluster's central processing units (CPUs) are cooled with spray cooled cold plates that have replaced the traditional air-cooled heatsinks. The cooling fluid used in the cluster is perfluorohexane, a perfluorocarbon (PFC) liquid long used in immersion cooling applications. Some of the key objectives for the effort are:

• Demonstrating the viability of a turnkey liquid-cooled commodity cluster.
• Demonstrating the ability to cool the CPUs while bypassing the chiller plant.
• Quantifying and displaying the real-time energy efficiency (Data Center Infrastructure Efficiency (DCiE)) and productivity for the data center.
• Demonstrating an environmentally sustainable data center cooling solution.

The cluster, named NW-ICE, passed a 30 day compliance test with 99.95 percent overall system uptime. Since the compliance testing, the liquid-cooled commodity cluster has run without liquid-cooling related incidents. A number of tests have been run to-date, which have demonstrated the ability to cool all the CPUs without chilled water (using 30°C, non-chilled water). To quantify the energy efficiency of NW-ICE at the facility level, PNNL has developed a comprehensive data acquisition, reduction, and display tool named FRED. FRED acquires power consumption data for all major mechanical subsystems and enables the calculation of the real-time energy efficiency of NW-ICE—the energy efficiency metric of choice is currently The Green Grid's DCiE (The Green Grid, 2007). Another effort with PNNL, SprayCool, and 3M involves the investigation of environmentally-friendly coolants. Like Hydro Fluorocarbon (HFC) refrigerants used in common data center cooling products, PFC working fluids are facing environmental scrutiny for their high global warming potentials (GWP). Low GWP fluids such as hydrofluoroethers and fluoroketones appear to be viable alternatives (Tuma, 2008). The presentation includes results from the liquid-cooled cluster implementation and long-term operation, the work towards demonstrating real-time energy efficiency for NW-ICE, and initial results from system testing with a low GWP fluoroketone.

Biography:

Phillip Tuma, P.E., is an advanced application development specialist in the Electronics Markets Materials Division of 3M Company. He has worked for 13 years developing applications for fluorinated heat transfer fluids
in various industries, including military and aerospace electronics, supercomputers, lasers, pharmaceuticals, and semiconductor manufacturing. Mr Tuma received a BA from the University of St. Thomas, a BSME from the University of Minnesota and a MSME from Arizona State University.

Back to Agenda