Increasing the ambient temperature in data center computer rooms to 65°C (149°F) can greatly reduce cooling. Operating at this temperature poses challenges, however, not the least of which is how hostile the environment is to personnel. The fans are placed at the rack intake to prevent them from operating at a temperature higher than 70°C (158°F). With this arrangement, the fan intake has to be designed so it can accommodate a dust filter. The exhaust setup is important to prevent a large speed variation when the air approaches the card cage, because this variation makes it difficult to meet the requirement that any card can be placed in any slot in the rack.
We designed a rack for one of our customers that could run indefinitely at a 65°C ambient environment and was able to dissipate up to 2 kW. The goals of this design were to reduce the pressure drop and the frequency of dust-filter maintenance, as well as to provide relatively uniform and well-distributed air flow so that higher power cards could be located in any slot.
High temperatures in a data center not only create potential operational problems but also cause issues with access and maintenance. Although 65°C is significantly lower than a sauna temperature, the comfort and safety of people is a serious concern. The rack design must allow for fast replacement of all components, including fans and filters. In addition, the maintenance frequency must remain the same or be reduced. Replacing fans more frequently increases cost and down time.
Managing Air Flow and Pressure Drop
Most fans are limited to 70°C during operation. At 65°C, the “mean time between failure” does not really allow an acceptable replacement frequency. A higher air-flow rate could reduce the temperature before the air goes through the fans, but it would add noise, and higher-performance fans would be needed to compensate for the increased pressure drop.
The practical solution is to install the fans at the intake of the rack. The air flow is determined only by the load of the rack, and the fans always face a maximum temperature of 65°C.
Implementing a design with fans at the intake requires management of the dynamic pressure drop. The dynamic pressure is related to the air velocity at the exit of the fans. If we had the freedom to place critical components in the high-dynamic-pressure regions, the dynamic pressure would be useful. But our customer’s specification was to generate uniform air flow for each slot. This air flow also had to be uniform from the front to the back of each slot. Airflow uniformity simplifies the design of new cards and the placement of new cards in the chassis. We had to manage the dynamic pressure drop in a volume as small as possible while maintaining a low pressure drop so that high-performance, high-cost fans would be unnecessary.
Keeping the Chips Cool
Going up to a 65°C ambient affects chip cooling. We considered two options to combat this issue: increasing the air flow and optimizing the heatsinks. We focused on pressure-drop optimization of the heatsinks to increase the flow going through them without pushing the fans to their limit and without generating a high pressure drop across the card cage. So we decided to maintain an air flow that was equivalent to the air-flow rate in the previous-generation rack.
Organizing the Cooling System
The system was organized so that it was compliant with the standard of the rack supplier. Figure 1 shows the sketch of the final design. Air intake is at the bottom front, and air exhaust is at the top back or top sides. Flow inside the rack is from bottom to top. Minimum air flow is 0.2 cubic meters per second (m3/s) or 420 cubic feet per minute (cfm). This air flow was defined so that this system would be backward-compatible with existing cards and racks. The size of the rack was fixed at approximately 0.4 m x 0.44 m x 0.25 m (H x W x D), which meant the air in the system would be replaced about four times per second.
Positioning the Fans
We implemented a horizontal fan-tray design because it allowed for a sufficient number of fans to be installed so that the minimum air-flow requirement could be met. The fan tray houses eight 92 mm x 92 mm x 38 mm fans. While a vertical orientation would allow for enough space behind the fan to easily manage the dynamic pressure drop (250 mm), it only would allow for placement of four side-by-side fans. Increasing the number of fans would require increasing the chassis height by 92 mm (close to 4 inches). Our analysis showed that it would be possible to manage the dynamic pressure without adding the additional plenum height.
Fitting the Air Intake
Because we had to limit the height as much as possible to increase the computing density, we needed to determine how much the height of the air intake could be reduced without affecting the pressure drop of the system. We simulated different heights of the air intake to draw a pressure-drop curve versus the height (Figure 2). When the derivative of this curve is close to zero, it means that increasing the height no longer significantly reduces the pressure drop. We selected the height of 50 mm (2 in.), with the option to increase the height of the intake to decrease the related pressure drop if needed.
Dealing with Dust
At the start of the design process, we had hoped to use the filter as a tool to manage the dynamic pressure drop by placing it at the exhaust of the fans. But we realized that the dust would rapidly clog the surfaces facing the fan blades because of the dynamic pressure. As a result, the air flow available to cool the electronics would change with time.
A vertical orientation of the filter at the inlet was not an option because of the high air speed resulting from a relatively small or reduced cross-section of the inlet. The pressure drop through the filter would require high-performance fans. In addition, the filter would clog more quickly because of the reduced filter surface area. In our simulation, we focused on the pressure drop and distribution of the air through the filter.
Our customer for this project chose the Mentor Graphics FloTherm 3D computational-fluid-dynamics (CFD) simulation software for air-flow and thermal simulations. There are multiple possibilities to simulate a filter, such as using collapsed resistances. Capturing the complex flow close to the filter becomes even more critical when the filter is in the range of 0 to 25 mm from the fans, so we used a thick resistance model with the dimensions of the 0.25-inch Quadrafoam filter we had selected. We also applied an advanced resistance model provided by the filter supplier.
Keeping the filter parallel to the fan was an option with one main advantage: the extraction of the filter is extremely ergonomic. We had to add 25 mm to the height of the system, and the dimensions of the filter were limited to the depth and width of the system. Increasing the surface should reduce the pressure drop and increase time between maintenance. Using the diagonal of the air intake volume increases the surface. This concept is used often in air conditioning systems. We confirmed this positioning, shown in Figure 3, was the best with simulations.
We wanted to make sure that the inclined filter would help the air to make a turn inside the intake area. We also wanted to confirm that all of the surface would be used uniformly to retain dust. The simulation showed that the pressure drop was very low (Figures 4‒6). The working pressure for the fans was average 95 Pa (minimum 86 Pa, maximum 106 Pa). We obtained the same results as if the filter was implemented horizontally with a free space of 25 mm in front of the fans, but without adding 25 mm to the total height of the system. The air approached the inclined filter at a uniform speed so the entire filter surface was evenly used. The inclined position of this filter still allowed fast extraction for maintenance.
Working Around the EMI Shield
A honeycomb electromagnetic-interference (EMI) shield that is 6.35 mm thick (0.25 inches) sits in front of the card tray. The EMI shield partially straightens the air flow (Figure 7). Unfortunately, it is made from a partially closed cell material that also restricts the airflow. Blowing air in a closed volume created a pressure drop (like in an interior dead flow area), where we observed vortices and flow disturbances that were difficult to simulate.
Before creating a component to make the flow for each slot uniform, we decided to look at the simulated flow, without considering the swirl. We observe that the speed variations are still very important at a distance 75 mm away from the fans (Figure 8).
We created a perforated plate that has two main functions: 1) stopping the flow where there are no slots to avoid blowing air into the closed cells of the EMI shield, and 2) reducing the flow in the region where we observed higher speed. This perforated flow distribution plate (FDP), shown in Figure 9, had to be in contact with the EMI shield. It would thus be closed at the entrance to the honeycomb cells already closed on the other side by the card rack. Because the plate does not require maintenance, it can be attached to the EMI shield instead of the fan rack. The distance between the fans and this perforated plate is critical. The flow must be able to be distributed through the open slots without creating an unacceptable pressure drop.
We plotted a curve using the data from three simulations of pressure drop versus distance (Figure 10), which indicated that 30 mm (1.2 in.) was a good compromise of space versus pressure drop with a good flow distribution. The plate did not generate a fully uniform flow (Figure 11), but the pressure drop created by the cards improved the distribution. The air-flow measurements made on the final and global simulation at the intake of each card slot showed an acceptable distribution from both left to right and from front to back.
Increasing the data center’s room ambient temperature to 65°C can be done by installing fans at the intake. We used a solution already well known in air conditioning to define the position of the dust filters, and we demonstrated by simulation that this solution can reduce the chassis volume. We used an isotropic advanced filter model in the CFD simulations to optimize the filter position close to the fans. The design also implemented a simple plate, with controlled flow resistance in areas where the dynamic pressure is too high. The result of these optimizations allowed us to create a rack where the flow was uniform for all card slots. The total height of the air intake and fan rack was just 120 mm.
The maintenance operation time was reduced to its minimum with all parts being easily accessible from the front. It was also possible to extract the fans and filter together to further reduce the maintenance time. Each fan was kept below 20% of its maximum throughput to avoid early fan failure and maximize equipment lifetime. All the surface area of the filter is used equally, avoiding any change in the air-flow distribution over time as the filter becomes clogged.
About the Author
Guy Diemunsch is a thermal expert at Institut Vendecom in France. His interest in thermal design started while he was preparing his PhD in physics from the Univesite de Franche-Comté UFC. In 1994 he joined Hewlett-Packard to start a new division developing high-end cooling solutions for workstation processors. In 2004 Guy joined in the power-electronics industry to solve the issues related to the increasing power densities of inverters and converters. MGE UPS, a Schneider Electric company, employed Diemunsch to develop a new air solution manufactured by Aavid Thermalloy. Guy joined Electronic Cooling Solutions in 2013. Since 2015, he has been creating solutions for hybrid and electric vehicles, including how to extract the heat from high-power-density electric engines.