The Detroit Water and Sewerage Department (DWSD), completed a $6 million data center modernization project that consists of highly automated standardized infrastructure from which applications and data will be deployed and provisioned based on workload demand. Power and cooling, the physical backbone of the data center, was designed and constructed to be flexible enough to keep up with automated, virtualized, dynamic technologies while balancing capacity constraints, efficiency demands and budgets. In today’s threat environment, protecting servers and information assets in data centers is critical. Security—both physical and cyber—were assessed and redesigned, and new systems were put in place to materially increase the department’s security posture in this area.
IT operations are a crucial aspect of most organizational operations around the world and at DWSD. One of the main concerns is business continuity; DWSD relies on the information systems to facilitate operations and customer services. If a system becomes unavailable, DWSD’s operations may be impaired. It was necessary to provide a reliable infrastructure for IT operations to minimize any chance of disruption. As information security was a major concern this data center upgrade had to offer a secure environment, minimizing the chances of a breach. The data center must therefore keep high standards for assuring the integrity and operation of its hosted computer environment, which was accomplished through redundancy of mechanical cooling and power systems, including emergency backup power generators and fiber-optic cables.
The purpose of the project was to upgrade DWSD’s data centers to industry standards for improved reliability and uptime, operational sustainability, and supportability. DWSD operates two data centers: a primary facility located and a backup facility at another location. The contracted work was performed to adhere to current Homeland Security policies for critical infrastructure, including approved environmental systems and cooling for critical data centers. As a result of the increased demand for computers, data and telephones, the department had to develop a reliable, secure and available communications infrastructure to its mission. The previous conditions of these data centers made them difficult to maintain, improve and troubleshoot; redundancy was minimal for critical applications. The overall object was to create a cleaner, safer and more organized data center environment so that DWSD could maintain and service the equipment, facility and users in a more cost-effective manner. The result allows DWSD personnel to function more efficiently during future relocations, additions and changes, reducing downtime. Other benefits of this project included making DWSD data centers compliant with Critical Infrastructure Protection (CIP) requirements, greater energy efficiency and the foundation of a system that can support future growth at lower costs.
Data center infrastructure management (DCIM) is the integration of information-technology (IT) and facility-management disciplines to centralize monitoring, management and intelligent capacity planning of a data center’s critical systems. Embracing DCIM aims to bring consistency, predictably and control to IT operational metrics while also improving service assurance.
These benefits were achieved through the implementation of specialized software, hardware and sensors; DCIM enabled a common, real-time monitoring and management platform for all interdependent systems across IT and facility infrastructures. This DCIM product will help the data center manager identify and eliminate sources of risk to increase availability of critical IT systems. It will identify interdependencies between facility and the IT infrastructure to alert the facility manager to gaps in system redundancy and provide dynamic, holistic benchmarks on power consumption and efficiency to measure the effectiveness of “green IT” initiatives. It’s important to measure and understand data center efficiency metrics; energy as well as server, storage, and staff utilization metrics will contribute to a more complete view of DWSD’s enterprise data centers.
Because the DWSD data center architecture spans multiple facilities, many challenges and design considerations had to be overcome to complete the project ahead of schedule and within budget. A detailed design process was undertaken once the appropriate conceptual design was complete. This phase included the detailed architectural, structural, mechanical, cabling technology, and electrical information and specification of the facilities. Modeling criteria aided in developing future-state scenarios for space, power, cooling and costs to allow for efficient use of the new mechanical and electrical systems and also to enable growth in the data center without needing to add new buildings, equipment or further upgrades of the incoming power supply.
Technology-infrastructure design addressed the telecommunications cabling systems that run throughout the data centers. The cabling systems for all three data center environments included horizontal cabling, voice, modem and facsimile telecommunications services; premises switching equipment; computer and telecommunications management connections; keyboard/video/mouse connections; and data communications. These data centers contain a set of critical routers and switches that transport traffic between the servers to the outside world. Redundancy of the Internet connection was also provided by using two service providers.
Some of the servers handle the Internet and intranet services needed by internal users—for instance, email, proxy and DNS servers. Network security elements were also deployed: firewalls, VPN gateways, intrusion-detection systems and so on. An on-site monitoring system for the network and applications helps provide insight into hardware health, multivendor device support, automated network device discovery and quick deployment. Additional off-site monitoring systems provide a holistic view of the LAN and WAN performance. “The project manager and construction team worked extremely well with the IT operations team on the migration of the live systems to the new infrastructure; they understood the flexibility needed during the process to minimize outages and downtime,” said the IT Datacenter Lead, Mr. U. Okike.
This upgrade now gives DWSD the capability to optimize the equipment housed in these data centers, strengthen the network infrastructure and work with partners to integrate their data. The project reduced the PUE and created a culture focused on being green and efficient that provides the infrastructure for delivering digital business services. DWSD’s future challenges are to ensure that every asset is utilized optimally; to eliminate fragmented operations, tools and information; and to collate and analyze the metrics needed to bridge the data center divide.
These data center facilities are the heart of the departments’ electronic business infrastructure, giving life to a significant percentage of business activities. Because these mission-critical facilities are unmanned and remotely managed from the network operations center (NOC) at DWSD’s Central Services Facility, vulnerabilities need to be addressed and mitigated. Access controls will be important: the data center management team must know the vendors and internal and external customers; maintenance practices and procedures must be rigid, centralized and authenticated.
Another important task is to ensure that the IT division avoids becoming understaffed and overworked, as it’s expected to deliver uptime as high as 99.999%. This statistic has a powerful significance: virtual perfection is expected. Since the data center environment is complex, administrators must avoid technical shortcuts, which can take their toll on support procedures and compromise overall security.
Server virtualization has been a game-changing technology for IT, providing efficiencies and capabilities that just were impossible in a physical environment. DWSD can now plan, deploy and maintain a sound virtual infrastructure. The interest of the department is now less in bottom-line costs and more on extracting business value.
About the Author
Anil Gosine has been involved in the water/wastewater industry for over 10 years; he was previously involved in the industrial construction management sector. His current and previous roles cover instrumentation, electrical and controls engineering, and construction project management. He is currently the industrial control system (ICS) administrator at Detroit Water and Sewerage Department, where he manages and administers the department-wide controls systems, ICS network infrastructure and cybersecurity.