In support of a Base Realignment and Closure (BRAC) 2005 action, BAE Systems was contracted by SPAWAR Systems Center (SSC) Atlantic to procure, produce, install, and test the passive infrastructure of a 312-rack data center in a remarkably short timeframe of less than six months.
Typically, the construction of a comparable data center would require an approximate one-year schedule and $8 million budget. But through standardized work processes, a synchronized supply chain, visual management and just-in-time (JIT) production strategies, the team finished the $5.4 million passive infrastructure project on time and $400,000 under budget. Furthermore, it delivered a 99.92 percent quality success rate, equivalent to a 4.65 sigma value.
This effort was a result of the BRAC 2005 decision to close the SSC Atlantic Pensacola office and relocate to a newly constructed facility in Charleston, SC. This facility will also become a focal point over the next two years, when the Department of the Navy (DoN) consolidates dozens of data centers to improve efficiency and operational readiness, while defraying expenses for hardware, software and upkeep.
The DoN is envisioning significant savings by consolidating operations and reducing hundreds of data centers, an especially important measure because of budgetary pressures. This initiative allows the military to build a collective baseline for its highest levels of security architecture.
Data Center Installation
Securing larger amounts of data on fewer servers requires efficiency, consensus of effort, and expertise. Through the application of lean principles, the team overcame a number of obstacles, including firm scheduling, a building under construction, seismic requirements, design alterations and logistic demands.
The project included production design, procurement and production services necessary to integrate, install and test the following components:
- 44,000 labels
- 20,000 fiber optic circuits
- 7,000 Category 6 cables
- 2,000 pieces of passive electronic equipment
- 624 power distribution units
- 26 miles of copper
The process taxed the 25-30 technicians physically and mentally, as the required miles of copper cabling equaled the distance of a marathon. The number of copper terminations additionally totaled 110,912, and the project involved 282 miles of fiber cabling—roughly the distance from North Charleston, S.C., to America’s oldest city: Saint Augustine, Florida.
Furthermore, all racks, equipment, connections and labels were inspected, yielding a handful of minor discrepancies that needed correction. Only 150 defects were found in 179,975 opportunities, and most were due to easily remediable labeling inaccuracies. The other errors belonged to copper terminations. No defects were attributed to rack installation, fiber termination, power termination or ground wires.
The effort also included concerns related to classified and unclassified networks, as well as TEMPEST, a code-name assigned to compromising emissions that, if intercepted and studied, could reveal the data conveyed, received or processed by the data center. The sensitive nature of the work spoke to the quality, innovation and cooperation required in sustaining crucial data center operations.
Cost Savings and Innovation
Although it was initially perceived that only sole-source procurement could fulfill the system’s stringent requirements, the team collaborated to identify sole-source alternatives and meet specifications. The options introduced competition into the procurement process, enabling the team to purchase materials from the original vendor at a cost savings of $800,000 over the originally quoted price.
The savings proved beneficial, as the money facilitated implementation of an innovative solution for seismic requirements. Two months into the project, a new condition was introduced necessitating Class C seismic design and installation. To support the alteration, the team procured ISO-Base platforms and placed the racks on top of these disaster- and earthquake-proof devices. The nonintrusive technology exceeds Seismic Zone 4 requirements and ameliorates the need for any type of seismic bracing or cabling.
Now, as long as other peripheral systems retain power, the equipment can continue to operate and process data during and after major earthquakes. It’s especially important near Charleston, as the area in 1886 experienced a powerful intraplate earthquake reaching between 6.6 and 7.3 on the Richter scale. Because of the region’s past, experts estimate the chances of Charleston experiencing another large earthquake at 1 in 400 each year.
The team also avoided schedule failure, as an ordinary solution (bracing bulkheads to the overhead, for example) would have extended the time span by two months.
Synchronized Supply Chain
To ensure that production efforts met customer schedule, the team built a synchronized JIT supply chain by implementing a lean pull system. The process divided the value stream into three sections: material procurement, initial production and install.
The team also adopted methods consistent with the theory of constraints. With the building initially unprepared for integration, build efforts began at an offsite production facility, eliminating time and work space constraints. The team also developed a supply chain and operations management solution by creating a flow of inventory. Ideal batch sizes of 28 racks were jointly produced to minimize on-hand inventory and optimize square footage. The team unpacked the equipment, staged the 312 equipment racks, installed equipment into the racks, and discarded and recycled the packing material—a large portion of the integration work—at the offsite production area.
The racks were then transferred to the data center and mounted on the ISO-Base platforms. Following installation, the install team would immediately pull the next batch from the production area, triggering the production team to simultaneously transfer raw materials into the manufacturing area, and the material team to request replacement material.
Visual Management and Modification
The team’s creation of visual workspaces and production boards also had a marked impact on value. Quality sheets and work orders were attached to each rack, enabling micro- and macro-level reports of production status. At a glance, anyone could walk through the production area or installation site and understand production status and quality performance.
The 44,000 labels also were given an easy-to-use, point-to-point scheme showing elevation and coordinates in accordance with Telecommunications Industry Association/Electronic Industries Alliance (TIA/EIA) standards. The labeling system enabled data center technicians to understand exactly where each of the 30,000 circuits was physically routed.
Besides the addition of Class C seismic design and installation requirements, the project demanded two other major scope changes. Late design modifications due to egress issues made necessary the addition of exits inside the data center, as the team removed 14 equipment racks, or 105 patch panels during the final weeks. The second change required the introduction of the Secret Internet Protocol Router Network (SIPRNet), the DoD’s classified version of the civilian Internet. The team reworked its cabling schedule and produced new labels to provide a shielded, copper cabling solution. The labeling system and production reports facilitated quick removal and replacement.
High-Level Strategies
JIT training offered an additional return because the technicians were taught phase-specific, standardized work processes directly preceding the start of each stage. The team brought in outside experts or representatives from the equipment manufacturer to demonstrate and train staff on the manufacturer-preferred methods of installation. The hands-on training allowed staff to manage the project scope and meet strict industry standards for the testing and installation of copper cable and fiber.
Such training fosters focus and supports industry-wide best practices. Comprehensive step-by-step processes also speak to sustainability and the creation of proper practices and culture, which hints at the core of lean methodology.
Altogether, these high-level strategies permitted scalability and flexibility, directly leading to improved competency and results. Through implementation, the team delivered an indispensible solution at a breakneck pace of less than six months, trimming nearly 7.5 percent from the original budget.
The second phase of the project is expected to be completed in early 2012, covering the integration of 222 power, temperature and humidity monitoring sensors through 81 routing devices; general cleaning; and mapping of the information technology drops.
|
Item |
Defects |
Opportunities |
| Rack Installation |
0 |
312 |
| Copper Terminations |
45 |
110,912 |
| Fiber Terminations |
0 |
17,654 |
| Power Termination |
0 |
6,121 |
| Ground Wires |
0 |
312 |
| Labels |
105 |
44,664 |
| Total |
150 |
179,975 |
Six Sigma Calculation:
| Defects/Million |
833.4491 |
| Success rate |
0.999167 |
| Sigma Value |
4.65 |
About the Project
This data center project reflects the strategic mission of BAE Systems, a large defense, security and aerospace company with locations around the globe. The team finished the first phase of the project in early October. For additional information regarding BAE Systems’ data center initiatives, contact Program Manager Don Barnard at donald.barnard@baesystems.com and (843) 614-5147.
Photo courtesy of The Planet dedicated hosting.
Article updated February 23, 2012.
