What are the risks that may affect the availability of a data center
- 1. What are the risks that may affect the availability of a data center Availability of a data center means the maximum uptime that the operation of a data center work without any failure. Availability is determined by a system’s reliability and it’s recovery time. Understanding that the system downtime can cause major impact on business entities, it is necessary to know what are the factors that can impact on data center availability. Generally these factors can be divided into 4 and listed as below, • Nature • Human • Utility • Equipment Nature This factor is having one of the major impact on availability of a data centers. We can’t predict the nature of earth which may change any time and cause to a complete disaster. This will include tornadoes, hurricanes, flooding , earthquakes etc. Control against the natural calamities by humans are really less hence this can have a major impact on availability of data center. Maintaining data access in the event of a disaster can mean the difference between a company’s success or failure. So let us have a look at some of the incidents that were occurred in various companies and their data centers. • Lightning: They say lightning doesn’t strike the same place twice, but in 2015 one of Google’s European data centers was struck by lightning not once, but four times, causing errors in 5% of the disks responsible for Google Compute Engine (GCE) instances. Although the company restored many of the drives, an estimated 0.000001% of data stored in the data center was irrecoverably lost. While that might not sound like much, try telling that to the customers who were affected by it. • Hurricanes: According to National Geographic, 2017 was the most expensive hurricane season in U.S. history, costing roughly $200 billion. With their combination of high winds, storm surge, and heavy rains, hurricanes are one of the most dangerous natural disasters data centers must contend with. The sudden flooding
- 2. resulting from Hurricane Sandy in 2012 caused extensive data center outages in New York and New Jersey. These failures were made even worse by the fact that backup systems were located in the same geographic region and where knocked out by the same weather event. • Tornadoes: A devastating 2011 tornado ripped through several hospital buildings in Joplin, Missouri, one of which was a data center. While none of the data lost was mission-critical, that was only because most of the information stored there had been migrated to a new offsite data center just a few weeks earlier. Hospital officials noted that if the tornado had hit a month earlier, the data loss would have been catastrophic and rendered the hospital completely inoperable. • Flooding: Severe flooding in Leeds, UK caused a Vodafone data center to temporarily lose power during Christmas of 2015. While data loss was negligible, the power outage disrupted mobile phone service temporarily. Vodafone, of course, has a bit of history with flooding, having suffered one of the most infamous data center disasters when its Istanbul data center was devastated by flooding in 2009. • Earthquakes: So far, data centers have been lucky. Modern architectural standards and additional precautions (such as special enclosures and rollers for server racks) have gone a long way towards protecting data centers from earthquakes, even in high-risk areas. • The Unexpected…: Disaster planning is all about expecting the unexpected. Take, for instance, the squirrel that knocked Yahoo’s Santa Clara data center offline for several hours in 2010, or the truck that drove into a transformer feeding power into a Backspace data center in 2007. Human According to a survey conducted by Aperture Research Institute, human errors are behind 57.3% of all data center outages. The second most common reason was improper failover with 43.7%.
- 3. Above: Diagram from the Aperture survey. Let me tell you the another survey details as well, According Uptime Institute: 70% of DC Outages due to Human Error and not by a fault in the infrastructure design. Furthermore, “mistakes” that led to an outage can often be traced to a poor decision by senior management. The results from both the organization can be different due to the reasons that it may be conducted on different entities and different environment. As a summary of both of these surveys we can conclude that the DC outage due to human mistakes are really much higher than any other dependencies. Let’s take an example of human raised DC issues, • Activation of the emergency power-off (EPO) switch • Adjusting the temperature from Fahrenheit to Celsius • Pulling power cords out of equipment • Overloading a circuit • Not following standard policies or procedures To minimize the risk of the “human factor” affecting operations, it is important to have up-to-date documentation on everything connected to your data center and manuals on how different critical operations should be performed. Manuals and
- 4. documentation together with scheduled tests should help you avoid many of the problems and outages described in this survey. Utility In the case of a data center the major source of utility is the electric power that is drawn to data center from local providers(can be a government entity or private entity). The secondary utility for a data center would be the Diesel generators and UPS systems. All other mechanical parts related to data center is directly or indirectly depend on the availability of utility. An Uptime Institute survey finds the power usage effectiveness of data centers is better than ever. However it is also true that survey indicates that the power outages have increased significantly. The Global Data Center Survey report from Uptime Institute gathered responses from nearly 900 data center operators and IT practitioners, both from major data center providers and from private, company- owned data centers(you can download the report from above link). Even though we do prepare all equipment’s for redundancies there is chances that these machines may not work as expected at the time of any incidents. One of the incident that I can get you is that - Diesel rotary uninterruptable power supply (DRUPS) systems were implicated in power disruptions that in 2014 affected Amazon Web Services in Sydney, a former Telecity facility called Sovereign House in London, now owned by Digital Realty Trust, and the Singapore Stock Exchange. Disruption at Amazon was caused by what the company called “an unusually long voltage sag.”. If you go through these incident you will understand the root cause of outage is due to utility failure and subsequent machines failed to start. Some of the incidents that is reported in data center imminent failure is as below, • Generator fail to start. • Generator fails after X number of hours running. • Utility power partially fails(usually one of three phases- phase loss) • UPS fails to switch to battery • UPS fails to switch from battery to input power From these incidents we can all say that maintaining the periodic checks, preventive maintenance tasks are really important that would really help a lot to avoid the impact of failures.
- 5. Equipment As you know the data center infrastructure is a large collection of multiple equipment and success is depending on the efficiency of all these together. Any equipment related to electric, mechanical, cooling, networking , servers are having chances to fail on an unexpected timeframe. Whether it’s a server reaching the end of its five-year expected lifespan or a UPS backup battery dying before it should, equipment failure is one of the most common causes of data center outages. With today’s powerful data center infrastructure management (DCIM) tools, facilities can monitor the overall health of their own equipment as well as colocated assets. While it may not be possible to predict every failure, sophisticated algorithms can monitor equipment performance continually to anticipate when hardware is reaching the end of its lifecycle or is prone to break down. When these problems are identified, data center personnel can plan to switch out faulty or outdated equipment without having to take critical systems offline. With the right redundancies and backups and emergency spares, in place, even an unexpected failure can be managed without compromising network performance. Source : www.vxchnge.com & www.pingdom.com
- 6. Have a comment or points to be review? Knowledge is power and it increases by sharing. Feel free to comment.