Commercial electrical power can be lost due to downed lines, malfunctions at a sub-station, inclement weather, planned ‘rolling‘ blackouts, or in extreme cases a grid-wide failure. As a business continuity practitioner, I regularly explore if businesses have implemented any ‘lessons learned’ from the Northeast power ‘Blackout’ of 2003. The answers vary from ‘yes we did’ to ‘why – it may never happen again’.
It has been 10 years since this event caught most off guard in its reach and magnitude. This blackout was a wide-reaching power outage that occurred throughout parts of the northeastern and Midwestern United States and many eastern Canadian provinces (including Ontario) later in the afternoon on Thursday, August 14, 2003. While some power was restored within six hours, many business and residences were at loss for upwards of 48 hours. It was estimated that the Blackout affected an estimated 10 million people in Ontario and 45 million people in the United States.
The Blackout’s primary cause was assigned to a computer program glitch at a control room of the FirstEnergy Corporation in Ohio. Operators were unaware of the need to redistribute power after transmission lines were overloaded. The glitch had the potential to be a manageable operational procedure with minimal local power loss disruption, but it turned into a national widespread event affecting many consumers (business and residential) in the northeastern and Midwestern United States and eastern Canada. Even when power was getting restored, affected areas were requested to limit power usage until the grid was back to full power.
Critical infrastructure impacted at that time included the water supply (lost water pressure), transportation (railroad service was stopped north of Philadelphia and the New York City area) and Canada’s Via Rail, which serves Toronto and Montreal and suffered schedule delays. Airports were closed. Many gas stations were unable to pump fuel due to lack of electricity. Many oil refineries on the east coast of the United States shut down and were slow to resume gasoline production.
The Blackout impacted communications well outside the immediate area of outage. Cellular communication devices were disrupted. Landline telephone continued to work, although some systems were overwhelmed by the volume of traffic, and millions of home users had only cordless telephones dependent on house current.
It is well known through many studies that power failures and power surges are by far the most frequent (45%) cause of data loss in IT systems (Source: ContingencyPlanning). Fortunately, with proactive planning and implementation of cost effective strategies, you can protect your business and mitigate the impacts against power disruption.
Getting Started – Protecting Your Computers, Servers and Data
Computing technology requires a constant and uninterrupted feed of stable and “clean” electrical power, which is power that does not ‘surge’ or ‘fail’ at any time. Taking a proactive approach with power loss planning – like designing a system that mitigates the impact to operations – and protecting your computers, servers and data from service interruptions is simple if you follow the steps below:
- Inventory all electronic equipment to be protected.
This can be done by undertaking a business impact analysis (BIA) which will define which processes and underlying IT and other technology infrastructure must be preserved when power failure and surges occur. BIAs should also address all costs to your business when the power goes out, including quantifying the cost of lost operational data, lost sales (current or future revenue), and impacts to the ‘customer experience’ — who rely on products and service delivery for their business.
- Include locations and required wattage required for each piece of equipment you want to connect to uninterruptible power to protect needed equipment.
- Include electrical outlets, plug type, circuit information, and amperage.
- Decide what level of power protection is required to provide each piece of equipment
- Does this equipment need power protection (i.e. surge protectors)?
- Does this equipment need uninterrupted power? (If the equipment is critical to the operation of your business and needs several minutes to safely shut down, it should be connected to an uninterruptible power supply [UPS].)
- How long do I want my equipment to be able to run in case of a blackout? (The total power demand [wattage] needed to operate the needed equipment during a blackout and the length of time you need your equipment to be operational will define whether a UPS or emergency power generation [EPG] equipment is needed.
- Do I need software that will automatically shut down my computer and save my files in the event of a blackout? (Many UPS’s have software that will automatically sense a power outage and perform an orderly shutdown of a computer connected to it. It is strongly recommended to install automatic shutdown software for all network servers.)
- Develop appropriate mitigation strategies and implement Power Protection Equipment.
There are three key basic levels of Power Protection Equipment commonly used today. Understanding the differences will help you decide which level is appropriate in protecting your business and equipment.
Level 1 Protection: The Surge Protector
The Surge Protector is a device that shields computer and other electronic devices from surges in electrical power or transient voltage, which flow from the power source. The standard voltage for homes and office buildings is 120 volts. Anything over this amount is considered transient and can damage electronic devices that are plugged into an outlet. It works by channeling the extra voltage into the outlet’s grounding wire, preventing it from flowing through the electronic devices while at the same time allowing the normal voltage to continue along its path.
Level 2 Protection: The Uninterruptible Power Supply (UPS)
An uninterruptible power supply — also known as an uninterruptible power source, UPS, or battery backup — is an electrical apparatus that provides emergency power to a load when main (commercial) power is interrupted or fails. A UPS provides battery backup that aids in saving data by keeping computer systems running with no interruption in the event of a brownout, blackout, or overvoltage. UPS’s also offer protection from surges, spikes, and sags. Many UPS’s are sized to allow computers attached to them to run for 10-25 minutes. This is sufficient time for users to log off their computers, write unsaved data to disk, and perform an orderly shutdown of the operating system. For a small network server room, expect to pay between several hundred to a few thousand dollars for UPS’s, and tens of thousands for larger server rooms. Advanced UPS systems with unique software are able to ‘sense’ power disruptions and can issue shutdown commands to the operating system to safely shut down any computers connected to them, aiding in saving data.
Level 3 Protection: Emergency Power Generation Equipment
A longer term power supply strategy during the loss of power to the UPS is the Emergency Power Generator (EPG). The EPG is usually powered by gasoline or diesel fuel and can provide power for extended periods. In a small installation, a portable generator is placed outside your business and extension cords are run from the generator to critical equipment and portable lights. For more complex environments, or permanent installations, the generator is permanently mounted and connected to the main power supply for the building.
For most small businesses, the cost of emergency power generation equipment is prohibitive, costing several thousand dollars or more. Most businesses choose to implement uninterruptible power supplies (UPS).
In modern buildings, most emergency power systems are still based on generators. Usually, these generators are driven by a diesel engine; although smaller buildings may use a generator driven by a gasoline engine, and larger ones driven by a gas turbine.
It should be noted most typical building emergency power systems supply emergency lighting to allow for safe exit during building evacuations and for illuminating service areas such as mechanical rooms and electric rooms. Exit signs, fire alarm systems and the electric motor pumps for the fire sprinklers are almost always run by emergency power. If you need emergency power from such generating equipment to ensure normal operations during power outages, then it must be designed, installed, and tested to ensure it meets with your defined performance requirements. For example, hospitals use emergency power outlets to power life support systems and monitoring equipment.
If you forego building the required infrastructure to address power surges and outages to protect from IT systems data and performance losses, you can move your applications/business systems to “cloud computing’. While cloud computing and data storage is accessible to all sizes of companies, the choices are endless. The question remains which one is right for your organization? You need to consider speed, security, scalability, recoverability, compliance and price.
In addition, business interruption insurance coverage is designed to help businesses that lose revenue due to unexpected shutdowns or limitations of operations. Generally, coverage is designed to protect a business’s income flow rather than its property. Business owners should familiarize themselves with the types of events that might force them to close and know whether these events arise from perils excluded from their property and business insurance coverage.
We can all learn from the recent power outage events and power outages are becoming more frequent. Having a plan of action to protect your business assets is one of the most cost effective strategies available. Take the challenge and learn what to do before disaster strikes….you can’t predict an emergency but you can prepare for one.
Be ready for anything: download ‘Business Continuity Planning for SIP Trunking: Ensuring Critical Connections,’ a white paper from Allstream.