The growth of cloud-IT adoption continues unabated. Today’s landscape of cloud providers is dominated by a small handful of companies based in the U.S. and China that deploy company-owned hyperscale data centers. For the foreseeable future, these few will account for much of the growth in the cloud-IT market as many businesses decide to give up owning their own data centers and move their applications to cloud platforms. Figure 1 shows the projected growth in the number of hyperscale data centers through 2020, along with the percentage of servers expected to reside in those data centers. By 2020, 47% of all servers sold are expected to go to hyperscale customers. For legacy hardware providers such as HPE and Dell, this situation portends a narrower customer base having both a higher technical acumen and a closer attention to the bottom line. HPE and Dell will see their sales to hyperscale customers captured by “no-name” white-box vendors that give their customers exactly what they want and no more. One server configuration no longer fits all. Open computing is increasingly discussed as a path to lower costs and higher efficiencies.
Figure 1. Source: Cisco Global Cloud Index, 2015–2020, Synergy Research.
New network topologies yield higher bandwidth for both “north-south” and “east-west” traffic in the hyperscale data center. Facebook has open sourced the specifications for its commodity-based switch designs and shown to the world the many benefits of software-defined networking (SDN) and network functions virtualization (NFV).
For the hyperscale data center operators, innovations in power delivery and new approaches to cooling enable the rack power density in their facilities to increase while improving the efficiency of the overall infrastructure, yielding extremely low PUE. This article discusses the role that power infrastructure plays in enabling hyperscale data centers to operate as reliably and efficiently as possible.
Who Are the Hyperscalers?
The most powerful names on the web today are the hyperscalers that began in the 1980s and 1990s during the Internet boom. Microsoft turned its success with MS-DOS into the over $90 billion behemoth it is today, moving from packaged software to SaaS (Office 365) and PaaS (Azure), enabled by a global data center footprint. Amazon began in 1994 as an online bookseller. Today, it’s more than a $170 billion e-commerce juggernaut, as well as the world’s largest cloud-service provider. Facebook is the world’s largest social-media-based enterprise, having numerous large data centers around the globe. Alibaba is China’s homegrown answer to Amazon, and is expanding its data center footprint beyond China’s borders. Google began as a research project in the 1990s and quickly rose to become the largest search provider on the Internet. That company too relies on massive data centers in multiple geographies to deliver the quick performance everyone has come to expect when searching the web. Other large and rapidly growing hyperscale data center occupants include Apple, Tencent, Baidu and EBay.
Most builders of hyperscale data centers own their own infrastructure. Originally populated with hardware made by Hewlett Packard Enterprise, DellEMC and Cisco, hyperscale data center operators are moving to lower-cost, purpose-built, custom and “open” compute and networking solutions from Taiwanese and Chinese ODMs, such as Quanta, Inventec, Wiwynn, Foxconn, Supermicro and Inspur.
Figure 2. Source: https://www.google.com/about/data centers/gallery/#/all/2.
The Impact of Power Density and Cooling on Hyperscale Design
During the past 200 years, the data center industry has seen rack power density go up commensurately with compute and storage densities. More servers and more hard drives are in a single rack today than ever, following a scale-out approach rather than scale-up. Whereas a typical IT rack once consumed 1–3 kilowatts, today loads of 20–40 kilowatts are common in the cabinet. The U.S. National Renewable Energy Laboratory reports that 30-kilowatt racks are widespread today.
Google used 5.7 terawatt hours of energy in 2015 according to Joe Kava, who heads the company’s global infrastructure. “Data centers make up the vast majority of that,” said Kava, which is why Google has also taken a leadership position in procuring renewable energy for its cloud campuses.
New servers based on the latest chip technologies from Intel, AMD, Nvidia and Arm deliver more compute operations per second and operations per watt than ever. At the same time, the cost of data center real estate has gone up in most markets, causing operators to seek taller IT cabinets to use space more efficiently. Having a tall column full of IT gear generating heat leads to cooling challenges. The data center architect must early on commit to either air or water as the cooling medium of choice. Some facilities deploy both.
Efficient air cooling requires adherence to a variety of design tenets:
- Alternate the direction of equipment to create natural hot aisles and cold aisles.
- Use blanking panels in empty rack spaces to ensure no leakage of cold air into the hot aisle.
- Practice either hot-aisle or cold-aisle containment.
- Natural convection can move air through a well-designed facility—cold air falls to the floor and warm air rises to the ceiling. Thus, employ overhead chilled-air service to the cold aisles and either raised ceilings or chimneys atop the back of the racks to push/pull air through the IT systems.
Adiabatic cooling relies on the process of reducing heat through a change in air pressure cause by volume expansion. Adiabatic processes have been used in data centers to facilitate “free cooling” methods that make efficient use of water and electricity.
Liquid cooling is well suited to applications where the cabinet power and thermal density exceed the cooling capacity of air flowing at a reasonable velocity (up to a few hundred CFM). Liquid cooling comes in many forms: chilled doors, chilled shelves, direct spray cooling onto a chip, and so forth. “In the HPC world, everything will move to liquid cooling,” said Paul Arts, technical director of Eurotech. “In our vision, this is the only way to get to exascale. We think this is the start of a new generation of HPC, with enormous power. We are just at the beginning of the revolution.”
Ultimately, the cooling decision will depend on the parameters under which the data center will operate: frequency of hard changes/repairs, number of available skilled personnel, how much time is allowable for repairs, the desired level of power efficiency and so forth.
Powering the Future Needs of Hyperscale Data Centers
Numerous papers document the growing power demands of the data center industry during the early 2000s. Had Moore’s Law and server virtualization not helped the industry, the growth in power demand would have likely exceeded 10% of the U.S.’s total production capacity. Instead, the IT industry took a leadership role in policing itself to make IT systems more power efficient. The power draw per server chip (CPU) was capped by Intel and AMD, and with each successive CPU generation, they improved the instructions per cycle and reduced the watts per instruction while remaining within a given power envelope.
Figure 3. Source: U.S. DOE, Lawrence Berkeley National Labs. SP = Service Provider.
Looking ahead, many hyperscale data center operators are committing to power their facilities with renewable energy. First achieved by purchasing renewable-energy certificates and offsets (buying green energy, such as hydroelectric, from local utilities), many hyperscale data centers today use energy generated on site by fuel cells or solar arrays. Apple and Facebook already have multiple locations running on solar. Microsoft has a biogas facility in Cheyenne, Wyoming; purchases wind power from Pilot Hill Wind; and generates solar energy at numerous facilities globally. Amazon built a 253 MW wind farm in Texas, and Google ran all its data centers on renewable energy during 2017. “We are the largest corporate purchaser of renewable energy in the world,” said Joe Kava, Google’s SVP of Technical Infrastructure.
There are numerous proposals across the IT industry regarding what type of power infrastructure is best suited to the hyperscale data center. Early on, Facebook chose 480V/277V AC power to the rack and 12V DC power to the IT loads in the rack. In recent contributions to Open Compute, Google proposed 48V DC to power the servers, with direct-conversion point-of-load power supplies running on 48V to the working voltages and currents that the motherboard needs to run the CPU, memory and so forth. In Google’s implementation, the power going to the rack from the power mains is 480/277V AC (or 400V/230V AC) with a three-phase rectifier on the power shelf converting AC power to DC power in the rack. Microsoft’s most recent Open Compute contribution has 480/277V AC to the rack and 277V into the server, with dual three-phase power supplies in the server enclosure and a basic (nonintelligent) PDU that supports blind mating to the servers in the rack.
For companies seeking power-generation equipment to run their data center, producing DC power on site and delivering it to the rack can make good sense. In that circumstance, bringing 380V DC to the rack and running a DC-to-DC converter to 48V is an option. Or it could even bring 48V to the rack, depending on distance from the power source.
Every data center is built to achieve function, latency, uptime, cost and value goals. The choice of power-delivery infrastructure is closely tied to these goals, mitigated by the robustness and flexibility of the software stack running in the data center. Future hyperscale data centers are likely to move from virtual machines to containers and to delivering “serverless” computing models. The debate over AC versus DC power to the rack will continue, while new technologies such as fuel-cell on rack will grow in popularity, and sourcing renewable energy to power the hyperscale data centers will become mandatory for the owners and operators. Learn more about the power of hyperscale computing here.
About the Author
Marc Cram is director of sales for Server Technology (STI), a brand of Legrand. Marc is driven by a passion to deliver a positive power experience for the data center owner/operator. He brings engineering, production, purchasing, marketing, sales and quality expertise from the automotive, PC, semiconductor and data center industries to give STI customers unequaled support and guidance through the journey of PDU product definition, selection and implementation. Marc earned a BSEE from Rice University and has over 30 years of experience in the field of electronics. Follow Marc on Twitter at @MCram01.