In today’s interconnected world, we send or post approximately 168 million emails, 11 million instant messages, 98,000 tweets and 695,000 Facebook updates every 60 seconds. Alongside the data transfers created by people, IoT applications will generate a further 3.9 exabytes of data by 2017 as a result of machine-to-machine communication.
The Data Explosion
All this Internet activity creates in excess of 1,820 terabytes of new data every minute, which has to be stored, processed and shared between a burgeoning number of data centers located across the world. Without the data center, there simply is no cloud.
The Internet has grown by about a factor of 100 over the past 10 years. To accommodate that growth, we have had to increase data center compute capacity by a much greater amount—about a factor of 1,000. To meet future demands on the Internet over the next 10 years, we will need to increase capacity by the same amount again. Currently, nobody really knows how we will get there.
Today, operators are thinking big and looking to mega data centers to provide the capacity we need. Mega data centers will make more use of software to define the infrastructure and take advantage of open architectures for both software and hardware.
But the industry has serious concerns about the viability of scaling present-day data center architectures to provide the capacity that we need. According to James Hamilton, vice president and distinguished engineer for Amazon Web Services, we are on “red alert” for the future of the data center.
Networking: A Perfect Storm
One can simplify the data center into two constituent parts: the server (the compute function), which performs data processing and storage, and the network, which interconnects the vast number of servers (typically 100,000+) in a mega data center.
Moore’s Law, still alive and kicking, has enabled microprocessor manufacturers to double the number of transistors on their chips every two years. The benefits of cheaper silicon propagate up to the server, allowing higher-performance machines and more storage to be built at less cost—factors that have been instrumental in driving the growth of the cloud.
Unfortunately, the benefits derived from Moore’s Law in relation to the compute function don’t fully apply to the networking part of the data center. For instance, data throughput, which is an important metric for the network, is determined by transistor speed, the number of physical pins available on a chip and, increasingly, aspects of new fiber-optic transport technologies—none of which is helped by Moore’s Law.
Consequently, while silicon is getting cheaper, networking costs are rising, and this situation is compounded as the server count grows. In the face of the anticipated scaling required, data center operators are now focusing on the network component.
A large part of the problem derives from the small size, or radix, of the basic network element: the electrical CMOS switch chip. A data center with over 100,000 servers that must enable any-to-any server communication requires an immense network interconnected through a vast array of CMOS switching chips that are constrained to 24 or 32 ports each.
Owing to the port-count limitations of these CMOS switch chips, vast numbers of network switch nodes and interconnections in the data center serve solely as intermediary connections between other switching nodes in an incredible spider-web-like network fabric. These fabric switching nodes and their interconnections are an expensive, power-hungry necessity dictated by the small size of the CMOS switch chips.
The radix of the basic switch element has a direct impact on the total number of switching nodes required in a data center of a specified size. The size of the radix dictates how individual switches and the data center as a whole are constructed (cost and power requirements), and it limits the scalability on the basis of cost and complexity. Connecting vast numbers of servers now takes a huge network fabric, and, even worse, the number of intermediary switching nodes and interconnections required grows as a multiple of the server count.
Spiraling data center costs and power consumption are limiting factors that have been well documented and are serious issues. It is the fundamental scalability of data centers, however, that is under question from this perfect storm in networking. Will it be possible to build data center networks to the size needed in the future?
Reinventing the Data Center Network
To enable future data centers, we need to significantly simplify the network. Fortunately, the industry is already moving toward simpler and more-efficient network architectures. One example is the move from classical single-rooted tree topologies with bandwidth bottlenecks and single points of failure, to folded Clos-based topologies that increase the network capacity by providing redundancy with multiple paths through the network.
Software-defined networks allow architects to separate the application, control and physical transport layers and to move them from proprietary hardware to open software. This approach allows the control-plane processing, which guides packets to their destination, to be performed on a set of commodity servers, thus simplifying the design of the switches.
These are both examples of important but fundamentally incremental improvements. To enable real scalability, we need much more. We need to deploy innovative, disruptive technologies in critical areas of the network.
We need three basic functions to create a data center network: packet processing, switching and transport. Today, the packet-processing and switching functions are implemented in electronics—typically, conventional CMOS technology—while the transport function is moving toward the higher bandwidth capability of optical (photonics) technology.
As we have seen, the current reliance on relatively small-radix switches created in CMOS severely constrains both the throughput and scalability of an economical mega data center network. But simplifying the work that the switch has to do by changing the network architecture gives us a unique opportunity, and we can now introduce optical technology into areas of the system that have previously been out of bounds for photonics.
This “smart integration” of electronics and photonics will enable each technology to play to its strengths: CMOS’s density and ability to perform complex processing, and photonics’ outright speed and transmission capacity.
Relaxing the constraints on switch size by dramatically increasing port density is fundamental to scaling data center capacity, simplifying the network and increasing throughput. A combination of electronics and photonics will enable packet switching in the optical domain to support scalable switch functions that are impossible in the electronic domain.
For example, high-speed data signals can travel much further over single-mode fiber optics than copper wire, and at much lower power and cost. This capability gives network designers more options to partition a large optical switch over multiple locations in a data center.
In addition, we can easily mix multiple signals over the same optical fiber using wavelength-division multiplexing, which reduces cable complexity and cost.
Towards Larger-Radix Switches
By integrating advanced optical switching features with the packet-processing capabilities of CMOS, we are able to achieve a larger radix for the basic switch element. Taking a modular approach to large, multistage switches simplifies their design and enables easier and cheaper connectivity for data centers.
This integrated optical switching solution will benefit from Moore’s Law scaling in CMOS technology, as well as advances in the optical domain. These two factors will help redirect the cost trajectory of networking to better align with the compute function and ameliorate data center operators’ concerns about how to increase data center capacity while keeping cost and power in check.
Combining CMOS and photonics in switching has the potential to reduce networking cost and power by a factor of 10. Bringing the switch function into the optical domain means it will not only scale with the growing size of the basic element but also, and uniquely, through the modular switch architecture. This approach allows unprecedented scalability in the network and the means to deliver the data center bandwidth capacity required for the next generation.
Developing this technology, however, will take a system-level multidisciplinary approach, which is why we are bringing together world-class experts in photonics, electronics, software, system architecture and semiconductor/photonics manufacturing.
Data center networking is a $10 billion and growing market. Silicon-photonics is the key to fulfilling this market potential and enabling the factor-of-1,000 increase in compute capacity that we will need to fuel the next 10 years of Internet growth.
About the Author
Dr. Andrew Rickman, OBE, is chief executive of Rockley Photonics and a pioneer in the field of photonics. He founded Bookham Technology in 1988 and took the company through a successful IPO in 2000.