Hurricane Sandy & Data Center Disaster Recovery
Hurricane Sandy takes data centers offline with flooding, power outages. Hosting customers stranded as generators in NY data centers run out of fuel.
Flooding and power outages caused by Hurricane Sandy have forced several New York data centers to switch to generator power. But those generators are quickly running out of fuel, so data center companies are telling their customers to shut down their servers and move workloads elsewhere.
One of the worst situations is at 75 Broad Street in Manhattan, where both Internap and Peer1 Hosting are shutting down operations "after basement-level flooding disabled critical diesel fuel pumps," Data Center Knowledge reports. 75 Broad Street is part of the "Zone A" portion of the city that is under emergency evacuation orders, as is another data center operated by Datagram at 33 Whitehall Street. The Datagram outage led to downtime for popular websites Gawker, Huffington Post, and BuzzFeed.
Peer1's official network status page reported last night that it was running on emergency generator power. This morning, the company said "we have an estimate of 4 hours for the fuel left on our generators. Our techs and facility are continuously working to get emergency fuel delivery on time and was looking to set-up a temporary tank and pump since the basement is still flooded. In the event of not receiving the fuel on time, worst case scenario is we will have to gracefully shutdown the facility."
The worst case scenario has apparently occurred, as the latest update says, "We are going to implement a controlled shutdown of NY Data Center at 10:45 ET." (UPDATE: Peer1 reported good news just before 12:30pm ET. "The New York facility is still on generator power, sustaining longer than initially estimated," Peer1 said. "We will have the latest update on the remaining fuel available, along with the arrival of fuel replacement shortly.")
Internap is reporting much the same scenario. In a note to customers made public on Pastebin, Internap said, "The flooding has submerged and destroyed the site's diesel pumps and is preventing fuel from being pumped to the generators on the mezzanine level. The available fuel reserves on the mezzanine level are estimated to support customer loads for approximately 5-7 hours. Once this fuel supply has been exhausted the generator will no longer be able to sustain operation and critical customer power loads will be lost. The building itself is being evacuated and no remote hands support will be available to assist in any equipment shutdown."
Internap advised its self-service customers to shut down their servers immediately and is having its customer support team execute a "graceful" shutdown of servers for managed customers. Internap's cloud services are also being shut down, the company said. We've asked Internap for an update and will report back if we get one. But according to iT News, Internap sent customers a follow-up e-mail this morning that said, "Available fuel reserves on the mezzanine level are estimated to be nearly depleted and able to support customer loads for less than 2 hours. Once this fuel supply has been exhausted the generator will no longer be able to sustain operation and critical customer power loads will be lost."
UPDATE: Internap is reportedly out of fuel and offline, but it is trying to get more fuel to the building. As of 12:55pm ET, the Internap network operations center hotline was playing a recorded message that says the facility is "currently without power due to flooding" and that co-location and IP customers can expect "widespread outages." Later in the day, the company posted a blog saying the 75 Broad Street facility was still without power, but that Internap is trying to get its generator farm back up and running. "It is unclear how long it will take ConEd to restore utility power to the site, but we are preparing for the possibility of remaining on generator power for many days," Internap said. In addition to running out of fuel in secondary tanks, Internap said the flooding damaged "both our redundant fuel pumps and our generator fuel tank." Internap is coordinating fuel deliveries and pumps, and said it will have engineers "fabricate pipe to bring the fuel directly to the generators on the mezzanine level."
In addition to damage caused by flooding, New York power company Con Edison said it preemptively shut off electricity in part of Lower Manhattan last night, and reported today that substation damage and downed wires cut off power to many customers. Con Edison called it "the largest storm-related outage in our history."
As mentioned, Gawker, Huffington Post, and BuzzFeed have suffered downtime as a result of flooding at Datagram's data center at 33 Whitehall Street. As of this writing, the main Gawker sites are still offline (stripped-down versions are reachable at live.gawker.com), while the Huffington Post and Buzzfeed have gotten themselves back online.
Further outages included Steadfast hosting at 121 Varick Street in New York City due to "an auxiliary electrical failure," and Init7 at 111 8th Avenue, due to a data center power outage. Init7 operates IP backbone services. As a result of the outage, the company said to expect "possible routing issues from/to the United States." As it turns out, Internap also has servers at the 8th Avenue address, but they are operating under generator power and have enough fuel for several days.
It's not as if data centers didn't take any precautions. In advance of the storm, Data Center Knowledge reported that "data center providers in New York, Philadelphia and the Washington, DC area said they are testing and fueling up their emergency backup generators, preparing to maintain services during any utility power outages caused by the hurricane." The cloud storage provider Nirvanix allowed customers to move data out of its data center in New Jersey for free. And cloud service providers such as Amazon are closely monitoring data centers on the East Coast, stocking up on generator fuel, and having extra staff on hand.
As various non-storm-related outages at Amazon have shown, customers relying on hosting providers and cloud services may want to build systems that can failover across multiple regions. Ultimately, when a data center is in the wrong spot at the wrong time, even the most extensive preparations may not be enough to stay online in the face of a storm like Hurricane Sandy.