Ah, the glamour of working in IT.
That thought probably didn’t occur to the team at Pier1’s Manhattan data centre as they hauled buckets of fuel up 18 flights of stairs during one hellish, 48-hour period in September 2012.
Their Herculean efforts kept Pier1’s backup generator running so the data centre could keep operating during superstorm Sandy (technically, it wasn’t a hurricane).
Similar scenes of chaos unfolded at seven other New York City data centres affected during the storm. As water poured down elevator shafts at Internap’s Manhattan data centre and flooded the lobby with three feet of water, staff posted this dramatic message online: “Life safety is our number one priority. We are making plans to completely exit the facility.”
When those data centres went down, so did websites like Huffington Post and Gawker (the latter site was knocked offline permanently by a Hulk Hogan body slam in 2016). Even after Sandy subsided, Internap remained on generator power for 10 long days.
Have we learned anything from Sandy?
Not as much as you might expect. The Uptime Institute’s new 2018 report on climate change and data centres found that:
- 71 per cent of data centres are not preparing for severe weather events
- 90 per cent don’t think they need to have a plan in place to mitigate increased flood risk
- 45 per cent are “ignoring the risk of climate change disruptions to their data centres”
Pointing to specific incidents and research, the Institute says climate change is increasing the risk of four major threats to data centres:
Storms/Flooding: With 17 named storms and 10 hurricanes (including Harvey and Maria), 2017 was the worst storm season in a decade and the seventh worst of the past 165 years.
Lightning: The Institute cites one study predicting lightning strikes will increase by 12 per cent for every one-degree Celsius rise in global average air temperature. This September, a lightning strike at Microsoft data centres in Texas affected Azure, Skype and Office 365 services for three days. Lightning strikes have also caused data centre outages at Amazon (2011), the Singapore Stock Exchange (2014), Google (2015) and AT&T (2018).
Drought: Severe drought conditions can lead to a shortage of the water required to cool data centres so they don’t overheat. In the U.S. alone, data centres are expected to consume 660 billion litres of water per year by 2020, up from 626 billion in 2014.
Wildfires: Hotter temperatures and drought increase the risk of wildfires. Between January and July this year, 37,718 wildfires consumed 4.8 million acres in the U.S., the fourth highest area on record. (That doesn’t include California’s recent Camp, Woolsey and Hill wildfires.)
Data centre defence
While data centre managers can’t do anything to stem climate change overnight, the Uptime Institute recommends steps they can take to protect their infrastructure from natural disasters — some of which are downright low-tech.
Call in extra troops: Citing the exhaustion and hunger of staff who defended data centres during Sandy, the Institute suggests arranging for extra help (even from out of town) and food on-site as extreme weather nears. It even recommends booking hotels to ensure a safe rest for local staff, who understandably “have their own homes, their own families, their own things to worry about … first before the data centre.”
Be cool: Consider cooling systems that are less reliant on water. The Institute warns that setups like evaporative adiabatic cooling “could become a problem as water scarcity issues increase around the world.”
Get high: Since water always flows downward, move critical infrastructure out of the basement. Many of the data centre, generator and fuel supply systems flooded during Sandy were subterranean.
Stock up: Don’t just keep fuel on hand for backup power generators; store it relatively close to those generators.
Unless you enjoy hauling buckets of fuel up 18 flights of stairs, that is.