Cloud Expo - Flying Two Mistakes High
2 likes632 views
This document discusses the importance of planning for failures when building highly available, scalable applications. It uses the analogy of "flying two mistakes high" when piloting radio controlled planes to emphasize that systems should be designed to handle at least two failures without crashing. The document provides examples of how extra capacity is needed to maintain availability during failures like node outages, rolling upgrades, and unknown dependencies between infrastructure components. It stresses the need to thoroughly analyze all potential failure modes and ensure recovery plans are robust enough to handle compounding issues.
![27
EXAMPLE
How many nodes do we need?
©2008-16 New Relic, Inc. All rights reserved.
Right???
§ ceil[1,000 / 300] = 4 nodes
§ With four nodes, can handle our traffic
§ PLUS we have enough nodes that we
can lose one! We have redundancy!](https://image.slidesharecdn.com/cloudexpo-flyingtwomistakeshigh-160609184343/85/Cloud-Expo-Flying-Two-Mistakes-High-27-320.jpg)
Cloud Expo - Flying Two Mistakes High
- 1. Flying Two Mistakes High A Guide to Not Crashing Lee Atchison,Principal Cloud Architect and Advocate at New Relic, Inc. ©2008-16 New Relic, Inc. All rights reserved.
- 2. 2 Safe Harbor ©2008-16 New Relic, Inc. All rights reserved. This document and the information herein (including any information that may be incorporated by reference) is provided for informational purposes only and should not be construed as an offer, commitment, promise or obligation on behalf of New Relic, Inc. (“New Relic”) to sell securities or deliver any product, material, code, functionality, or other feature. Any information provided hereby is proprietary to New Relic and may not be replicated or disclosed without New Relic’s express written permission. Such information may contain forward-looking statements within the meaning of federal securities laws. Any statement that is not a historical fact or refers to expectations, projections, future plans, objectives, estimates, goals, or other characterizations of future events is a forward-looking statement. These forward-looking statements can often be identified as such because the context of the statement will include words such as “believes,” “anticipates,”, “expects” or words of similar import. Actual results may differ materially from those expressed in these forward-looking statements, which speak only as of the date hereof, and are subject to change at any time without notice. Existing and prospective investors, customers and other third parties transacting business with New Relic are cautioned not to place undue reliance on this forward-looking information. The achievement or success of the matters covered by such forward-looking statements are based on New Relic’s current assumptions, expectations, and beliefs and are subject to substantial risks, uncertainties, assumptions, and changes in circumstances that may cause the actual results, performance, or achievements to differ materially from those expressed or implied in any forward-looking statement. Further information on factors that could affect such forward-looking statements is included in the filings we make with the SEC from time to time. Copies of these documents may be obtained by visiting New Relic’s Investor Relations website at http://ir.newrelic.com or the SEC’s website at www.sec.gov. New Relic assumes no obligation and does not intend to update these forward-looking statements, except as required by law. New Relic makes no warranties, expressed or implied, in this document or otherwise, with respect to the information provided.
- 21. This same applies when building highly available, high scale applications 21©2008-16 New Relic, Inc. All rights reserved.
- 23. 23©2008-16 New Relic, Inc. All rights reserved. Walk through ramifications and recovery plan Make sure recovery plan works § Has no mistakes § Has its own recovery plan How do we keep “Two Mistakes High” in an application?
- 24. 24©2008-16 New Relic, Inc. All rights reserved. Walk through ramifications and recovery plan If recovery plan doesn’t work… it’s not a good recovery plan Make sure recovery plan works § Has no mistakes § Has its own recovery plan How do we keep “Two Mistakes High” in an application?
- 25. 25©2008-16 New Relic, Inc. All rights reserved. EXAMPLE How many nodes do we need?
- 27. 27 EXAMPLE How many nodes do we need? ©2008-16 New Relic, Inc. All rights reserved. Right??? § ceil[1,000 / 300] = 4 nodes § With four nodes, can handle our traffic § PLUS we have enough nodes that we can lose one! We have redundancy!
- 31. 31©2008-16 New Relic, Inc. All rights reserved. EXAMPLE Rolling upgrades
- 35. 35©2008-16 New Relic, Inc. All rights reserved. EXAMPLE Unknown dependencies ? ?
- 42. 42©2008-16 New Relic, Inc. All rights reserved. EXAMPLE Failure loop
- 43. 43 EXAMPLE Failure loop ©2008-16 New Relic, Inc. All rights reserved. Are you safe from power outages? You live in an apartment… § The apartment provides an enclosed garage to store things in § The power goes out in your place a lot… § ... you buy a generator, store it in the garage
- 44. 44 EXAMPLE Failure loop ©2008-16 New Relic, Inc. All rights reserved. Oops Oops… the garage: § Has a single door, the big garage door § It has a garage door opener § That requires electricity to open... § The generator is only available... when you already have power…
- 46. 46©2008-16 New Relic, Inc. All rights reserved. EXAMPLE High redundancy in action
- 50. 50 EXAMPLE US Space Shuttle Program ©2008-16 New Relic, Inc. All rights reserved. § They had problems… serious mechanical problems... § But the software system utilized state of the art: • Redundancy techniques • Error recovery techniques
- 51. 51 EXAMPLE US Space Shuttle System ©2008-16 New Relic, Inc. All rights reserved. Five onboard computers § Four were identical (fifth talk about later) § All four: – Ran the exact same program during critical periods – Given same data – Expected to generate the same result
- 55. 55 EXAMPLE Four computers ©2008-16 New Relic, Inc. All rights reserved. Could FLY with only THREE computers working Could LAND with only TWO computers working
- 57. 57 EXAMPLE Ties ©2008-16 New Relic, Inc. All rights reserved. What if the four computers couldn’t decide? (software bug or multiple failures) Fifth computer was used as a tie breaker § Much simpler version of software… only used for key decisions § Software written by independent software team, unconnected with rest of software developers § (In theory) would not introduce same software errors…
- 58. 58 Highly Successful ©2008-16 New Relic, Inc. All rights reserved. 30-year operation of Space Shuttle: § Never a case where a serious life threatening problem occurred that was a result of a software problem § Even though software was the most complex software ever built for a space program
- 66. ©2008-15 New Relic,Inc.All rights reserved. Thank you. Lee Atchison Principal Cloud Architect and Advocate at New Relic, Inc. Architecting for Scale Published by: O’Reilly Media, Available: June 2016 www.architectingforscale.com @leeatchison leeatchison