It can happen to anyone—everything in your facility is working as intended, and suddenly, without warning, all your company’s activities come grinding to a halt. It turns out that even though you did your job right, somewhere along the line, someone else didn’t. Somewhere outside your facility, a cable has been cut, and the end result is that your Internet service provider (ISP) is down. We have become very dependent on reliable Internet, so an outage has many harmful repercussions for a company and its clients.
And don’t think it’s something that could never happen to you. A cut cable brought Wikipedia to its knees back in 2012. A more than 40-minute 2015 Amazon Web Services (AWS) outage resulted from a route leak in a critical data center. A user on Tales from Tech Support even has an amusing, but not very happy, story of how a hospital’s outage was caused by a drunk taking a shotgun to an aerial fiber cable. Essentially, companies and individuals always need to be prepared for a technical glitch with their ISP.
Apparently, ISP outages are more common than you would think in rural areas. In the event that you’re hit with an outage caused by an unexpected incident, it’s essential that you have a clear, established response process that includes the following:
- Immediately contacting your administrator when you realize you’ve lost connectivity.
- Logging the incident with a clear description, including date, time and any other relevant information.
- Sending out a team to locate the source of the outage.
- Notifying any affected clients immediately when the outage occurs, and then again once you’ve determined the outage’s cause.
- Reaching out to clients through a variety of channels such as email and social media to notify them when the outage has been addressed and apologize for any inconvenience the outage caused.
Unfortunately, if you’re dealing with a cut cable, there is little you can do except try to restore service. That isn’t to say there are not ways you can prepare for such an incident, mind you. There are many active steps you can take to make your network more redundant. They include the following:
- Establishing multiple routes and ensuring that fiber running to and from your facility is fully fault tolerant.
- Implementing better Border Gateway Protocol (BGP) policies to prevent route leaks.
- Setting up agreements with multiple carriers, so that if one carrier is rendered unavailable, you can switch to an alternate carrier.
- Ensuring route switching can happen automatically.
Here’s the problem: most organizations simply lack the resources to establish this redundancy on their own. The costs add up rather quickly, after all. A customer operating a high-availability data center must have at least two different ISPs with two different physical paths into the facility. Inside, they would need to purchase larger routers and switches to accept multiple ISP feeds, then establish BGP policies to effectively monitor ISP health.
The construction costs alone would be astronomical, never mind the enormous cost of operation. If your business is operating its own data center, then you probably have the necessary resources to handle the added infrastructure. But if you’re a smaller organization with just a few racks, you’ll be unable to afford such a complex network environment—nor will you have what you need in labor and expertise to operate it.
That’s precisely why colocation is an effective choice, especially for high-availability applications.
In a colocation facility, you will generally have the option to choose between a wide range of independent carriers. Most colocation hosts also operate highly-redundant network architecture, including dedicated fiber trunks, high-end routers and switches, and multiple ISPs. Additionally, their circuitry tends to be larger than a customer might be able to purchase individually.
Either way, this issue is something you need to address, because the damages caused by ignoring any potential outage far outstrip any savings you might enjoy. The costs surrounding outages are significantly increasing. For example the Emerson Network Power reports that the average cost of an unplanned outage is now around $8,851 per minute, for a total of around $740,357 for each outage. Worse still, maximum downtime costs are rising sharply as well, with a current high of $2,409,911.
That’s no small amount of cash. And the cost isn’t just monetary, either. If you’re running a SaaS company, think of the clients you might lose as a result of downtime. There are customers who rely on your servers being available for their daily activities. It falls to you to do everything in your power to reduce downtime as much as possible, and if you lack the necessary capital to achieve that goal in your own facility, that facility is insufficient and you should consider alternatives.
“Today’s data center and cloud ecosystem have evolved to support a growing business, a dynamic workforce and constantly changing business demands. In this challenging environment, data center administrators are tasked with delivering the IT part of business continuity,” writes Bill Kleyman of Data Center Frontier. “[And] it’s clear that as reliance on the modern data center continues to grow—there will be even more demand around infrastructure uptime.”
Transit outages can be incredibly problematic, and any facility can be vulnerable to them, even those that do everything else right. By ensuring you have redundant networking technology and setting up an ironclad incident-response process, however, you can significantly reduce the risk that such events will happen to you. And if you don’t have the resources on your own, consider colocating in a data center facility that does.
About the Author
Tim Mullahy is the general manager at Liberty Center One. Liberty Center One is a new breed of data center located in Royal Oak, MI. Liberty can host any customer solution regardless of space, power or networking/bandwidth requirements.