At the end of the twentieth century, there was consensus that the previous couple decades were the dawning of the world’s information-technology (IT) age. Within the first few years of the twenty-first century, however, it has become clear that although the technology can be credited to the past century, the information defines the new one. After only 15 years, the amount of data that has been generated this century dwarfs the amounts of all the generations that came before, and all that data is open, available and instantly ready to be observed and analyzed.
This significant market trend has had a large impact on enterprises, which have started viewing information technology as a fundamental resource to their business processes rather than as a means to track business performance. Now companies must analyze not just business data but also other parameters, such as the authentic experiences of its customers and partners or the impact of upcoming events on projected sales (based on past experience). For example, a large retail enterprise that has embraced big data to optimize its digital market strategy is now moving on to optimizing even its supply chain. Data analytics can be used to optimize an omni-channel supply chain to ensure an efficient, streamlined and unified experience for customers, partners and employees. This approach allows retailers to ensure that the latest trends are detected and the correct stock is ordered and warehoused to maximize revenues and guarantee satisfied customers.
As a result of these new breakthroughs, enterprise IT organizations have had to react quickly to modernize data center infrastructure such that it is capable of responding to the very demanding requirements of these new decision-making workloads.
By building a robust infrastructure that can process the ever growing volume of data in near real time, organizations will have a head start in the “data analytics race.” This new infrastructure should be based on the most recent multicore and multisocket servers, combined with faster flash-based software-defined storage (SDS) and, most importantly, connected by high-performance software-defined networking (SDN)—an essential element for fast, secure data communication, eliminating potential bottlenecks, maximizing traffic flow and meeting the corporate criteria for total cost of ownership (TCO).
The Need for High-Performance Networking
Why is the performance of the network so critical in software-defined data centers (SDDCs)? The primary reason is that the SDDC is based on modern scale-out architecture, with networking to connect the commodity servers and storage. The faster the connectivity, the faster the entire cluster will work. A good example is the SDS systems, in which data can reside on any of the physical servers; the faster that data is delivered over the network, the higher the achieved efficiency.
But bandwidth is only one of the parameters that must be considered. The other important consideration is the efficiency of the network, which is measured by the percentage of CPU necessary to run the communications tasks. The more CPU cycles needed for communications, the fewer CPU cycles are available to run jobs. An efficient network can offload the communication tasks from the CPU to the I/O controller, eliminating I/O bottlenecks and resulting in a more balanced cluster.
One of the most popular technologies used by efficient networks is remote direct memory access (RDMA). Interconnects based on RDMA offer the ideal option for boosting data center efficiency, reducing overall complexity and increasing data-delivery performance. RDMA enables data to be transferred with minimum involvement from the CPU and without the need to copy that data multiple times, as is required when running a TCP/IP Ethernet stack (see Figure 1). RDMA not only frees expensive CPU cycles to run applications, but it also minimizes the overall data communication latency with low jitter, making the overall execution time more predictable. Until recently, though, RDMA was only available in InfiniBand fabrics. With the advent of RDMA over Converged Ethernet (RoCE), the benefits of RDMA are now available to data centers that are based on an Ethernet or mixed-protocol fabric as well (IBTA).
RDMA over Converged Ethernet (RoCE)
RoCE employs advances in Ethernet to enable efficient implementation of RDMA over Ethernet. It enables widespread deployment of RDMA technologies in mainstream data center applications. RoCE-based network management is the same as that for any Ethernet network, eliminating the need for IT managers to learn new technologies. The business advantages of RoCE for a data center include the following:
- No changes to data center infrastructure
- I/O unification on a single wire over 10/25/40/50/100GbE networks
- Continuation of the existing data center management infrastructure
- Reduction in power consumption, leading to cost savings
- Maintaining compatibility of existing and future applications
- Significant capex and opex savings with a single-chip solution for I/O unification
The RoCE interconnect has been massively deployed already in large cloud-service providers like Microsoft Azure, in enterprise data centers and in converged infrastructure systems. Also, at the Ignite’15 conference, Microsoft presented an overview of their Windows Server 2016 SDS solution, Storage Spaces over 100Gb/s Ethernet and RoCE. Four cores ran the transport protocol, and NVMe SSDs provided high bandwidth to the storage. The results clearly demonstrate the advantages that RoCE offers versus TCP/IP in Ethernet, enabling 2x higher throughput (92Gb/s versus 54Gb/s) and 50% less latency. Moreover, the results show a 2x improvement in CPU efficiency: when running the TCP/IP protocol, all four cores were fully utilized, whereas RoCE used less than 50% of the cycles, leaving more CPU cycles to run the applications and thereby enabling higher efficiency.
Figure 2: TCP/IP fully utilized all four cores that were allocated to run the transport protocol.
iSCSI Extensions for RDMA (iSER)
In addition to server-to-server communication over RDMA, the IBTA also standardized iSCSI over RDMA (iSER), a computer-network protocol that extends the Internet Small Computer System Interface (iSCSI) protocol to use Remote RDMA. It basically employs the upper layers of iSCSI for session management, discovery, recovery and so on; it’s therefore compatible with all the features and functions supported by iSCSI. Compared with software-based iSCSI adapters like VMware’s software iSCSI adapter, however, using iSER eliminates the bottleneck through the following mechanisms:
- iSER uses zero copy via RDMA technology
- iSER CRC is calculated by hardware
- iSER works with message boundaries instead of streams
- iSER transport protocol is implemented in hardware (minimal CPU cycles per I/O)
Figure 3: iSER enables higher throughput and more IOPS (ESXi 5.X, 2 VMs, 2 LUNs per VM).
RoCE and iSER interconnects are also behind Dell’s Fluid Cache for SAN solution, which uses a distributed cache mechanism to minimize access to the physical storage. The solution, which first demoed live at Dell World 2013, maximizes the cluster’s performance by enabling VM access to data stored in another VM’s cache. This task requires a high-bandwidth and low-latency interconnect.
The results were impressive. The solution reached 5 million random read IOPS (input/output per second). When running an online transaction processing (OLTP) Oracle workload benchmark, Dell Fluid Cache for SAN also achieved
- 99% faster response time
- 4x increase in transactions per second
- 6x increase in concurrent users
Dell has also seen significant performance improvements via Dell lab tests of a hardware stack enabled with Dell Fluid Cache for SAN running Microsoft SQL Server database on VMware software (http://marketing.dell.com/SQLServer-OLTP).
Efficient Networks Improve Return on Investment (ROI)
The performance boost that RDMA-enabled networks deliver has a significant impact on the data center ROI. In a benchmark that built a virtual desktop infrastructure (VDI) over 10Gb/s RoCE versus 10Gb/s Ethernet, RoCE was able to deliver 2.5 times the number of virtual desktops per server (140 vs. 60).
When doing an ROI analysis of running 5,000 virtual desktops, RoCE enables savings of $336,400 exclusively on hardware. Taking into account the software licenses and the opex, the saving can easily reach (and may pass) the $500,000 mark.
Figure 5: Savings of $730k when running 5,000 virtual desktops—assuming fully loaded server with SSDs at the cost of $6.8k and a switch with 48 x 10GbE ports at a cost of $10k.
Summary: RDMA Boosts Corporate Success
The ability to store and quickly analyze massive amounts of data has become a significant competitive force, reshaping data center and application architectures and transforming the way IT resources and applications are created, bought and managed.
Enterprise IT teams cannot ignore the technological challenges that are associated with building the right data center infrastructure to support these new mission-critical business processes. As such, it is imperative that they deploy the most efficient infrastructure capable of supporting the new requirements.
Building software-defined data center infrastructure over RDMA-enabled interconnects improves business performance while reducing IT operational costs, helping companies to gain a competitive advantage across all dimensions of their business.
About the Author
Motti Beck is Director of Enterprise Market Development at Mellanox Technologies. Before joining Mellanox, Motti was a founder of several startup companies including BindKey Technologies, which was acquired by DuPont Photomask, and Butterfly Communications, which was acquired by Texas Instruments. He was previously a business-unit director at National Semiconductor. Motti holds a BSc in computer engineering from the Technion-Israel Institute of Technology. Follow Motti on Twitter: @MottiBeck.