Hurricane Sandy Highlights Need for Disaster Preparedness

This week’s landfall of hurricane Sandy in the New York, New Jersey, Northern Virginia and surrounding areas is another reminder of the fact that businesses and even lives can be brought to a disturbing halt—and often, there’s little anyone can do to prevent the ensuing destruction. This most recent hurricane illustrates how even unlikely events can sometimes crop up. The data center industry, which has a major hub in New York City and is also concentrated in other areas on the eastern seaboard, had the unfortunate chance to put its disaster preparedness efforts to the test. Here’s a smattering of what has been taking place in the hurricane’s aftermath, from the darn near heroic to the downright annoying.

Good Old-Fashioned Manpower

Data centers are facilities that do well with maximum system redundancy and automation, greatly reducing the need for human interaction to keep them running. But sometimes there really is no substitute for human beings. Following Con Edison’s shutdown of power to Lower Manhattan and subsequent flooding from Sandy, a Peer 1 Hosting data center on the 17^th floor of a building in that neighborhood faced a dire situation. According to Computerworld (“Huge customer effort keeps flooded NYC data center running”), Peer 1’s “rooftop generator couldn't access the 20,000-gallon fuel tank in the flooded basement. Its pumping system was disabled by storm waters from Hurricane Sandy.” The solution? A bucket brigade moving fuel by hand up to the generator to keep the facility running. “A network of people, stationed at every staircase landing, formed a bucket brigade passing fuel from one person to the next. Several dozen may have been involved over multiple shifts, including some day laborers who were also hired to help.”

Although the effort to keep the Peer 1 data center operational might not quite meet the standard of heroic, it certainly is one of the more encouraging data center stories of the disaster. Other data centers were unable to stay online, necessitating reliance on backup facilities.

The Need for Backup—and More

The Peer 1 facility illustrates how a number of different problems caused by a disaster can combine to worsen the situation. Not only was electrical power cut off by the utility company (something that could happen purposefully, as in the case of ConEd, or as a direct result of damage from the disaster), but floodwaters cut off access to the backup generator’s fuel supply. In some such cases, no amount of onsite preparation can keep a data center going until normal services are restored.

DatacenterDynamics (“Datagram data center stays down through Wednesday”) notes that a Datagram data center in Manhattan was forced to shut down when the building’s basement was flooded, creating hazardous conditions because of the electrical infrastructure therein. In this case, a shutdown of the facility was the only option. “Crews continued pumping water out of the basement on Wednesday afternoon. Power to the building could not be restored until all water was removed, the company said on a status page it put up to replace its website.” The hurricane also damaged communications lines to the facility, adding to the difficulties. The solution, however, was to switch to an alternate site: “The provider has been offering backup servers to New York customers out of its facility in Connecticut.” Redundancy in location—as well as in onsite systems—enabled Datagram to continue providing services despite a complete shutdown of its facility in Manhattan. Other companies did likewise, using failover sites to maintain operations despite difficulties in Manhattan or other areas.

Of course, some companies faced less dire circumstances and were able to more easily weather the storms through their onsite backup systems, as DatacenterDynamics also notes (“Sandy: the uptime stories”).

Stock Markets Close

Perhaps wisely, the New York stock markets closed on Monday and Tuesday, finally reopening on Wednesday. The extent of high-frequency trading (also called algorithmic trading), combined with the effects of the hurricane could have triggered a flash crash or other anomaly—a serious danger, given the fragile state of the economy. Not only does the hurricane present a solemn reminder of the need for disaster preparedness, but it highlights the dangers that disasters (natural or manmade) pose to a nation on the brink of another recession. According to Computerworld (“Even with prep, did Wall Street's business continuity plans fail?”), an IDC Financial Insights report “criticized the NYSE's contingency plan…because the plan called for the stock market to operate ‘as an electronic-only exchange for the first time in its history.’” The report went on to say that “keeping the market open on a purely electronic basis, with the market having never operated this way even under perfect conditions, would only increase the chance of any minor malfunction to a high-frequency trading algorithm, causing potentially great disruption.”

Unfortunately, the report cited insufficient regulation as part of the problem. “‘Not only has Sandy exposed the susceptibility of the contingency plans devised by major market players (exposing a lack of foresight concerning the potential impact of a hurricane that can simultaneously hit the [New York-Connecticut-New Jersey] area), but it has also given insight into [financial market regulators’] apparent inability to monitor electronic markets,’” according to Computerworld. Presumably, this means that the solution is more regulation—an increasingly common theme in American society, even though the federal government is up to its ears in debt from meddling in the business of almost everyone around the world.

The Takeaway

For many business owners (and consumers), the images of Sandy’s destruction—and the stories of individuals and businesses attempting to continue their lives and operations—are too far removed. Unfortunately, however, disasters can strike anywhere. Few would have anticipated a hurricane as being much of a threat to New York City, for instance (compared with, say, another manmade attack). The importance of both redundant systems on site and a remote backup site has been abundantly illustrated. Of course, not every business can afford all the disaster-preparedness and -recovery stops: budget is still a limiting factor. But for those companies that can afford to prepare but simply won’t, hurricane Sandy should be a wakeup call. The costs of downtime often dwarf the costs of preparing for disasters, major or minor. And for everyone, this disaster is an opportunity to reflect on the importance of life—it’s more than just business and cool technology—as well as the path of the nation and even the western world. A vibrant economy can handle a disaster and bounce back quickly; a struggling economy can be damaged severely by the same disaster. Whatever the case, however, we hope for a safe and speedy recovery in the areas harmed by Sandy.

Photo courtesy of Brian R. Birke