5 Essential Techniques for Building Fault-tolerant Systems
4 likes6,304 views
The document outlines five essential techniques for building fault-tolerant systems: containment, fail fast, escape, adjust, and learn. It emphasizes the importance of anticipating faults, implementing measures to minimize their impact, and fostering a culture of continuous improvement through monitoring and learning from incidents. These strategies aim to ensure system resilience and maintain availability even in the face of component failures.
5 Essential Techniques for Building Fault-tolerant Systems
- 1. 5 Essential Techniques for Building Fault-tolerant Systems DIEGO BERRUETA | ENGINEERING PRINCIPAL | ATLASSIAN
- 2. Preston Rhea; Flickr (www.flickr.com/photos/prestonrhea/), CC-by
- 3. ALERT!
- 9. Fault-tolerant system Continues to operate in the event of faults in some of its components Fault Deviation from the normal state, either due to internal or external causes Failure Observable impact of a fault in a system, usually manifests as reduced availability
- 10. Fault-tolerant system Continues to operate in the event of faults in some of its components Fault Deviation from the normal state, either due to internal or external causes Failure Observable impact of a fault in a system, usually manifests as reduced availability
- 11. Fault-tolerant system Continues to operate in the event of faults in some of its components Fault Deviation from the normal state, either due to internal or external causes Failure Observable impact of a fault in a system, usually manifests as reduced availability
- 12. Bob Yeats (OSU); Flickr (www.flickr.com/photos/oregonstateuniversity/), CC-by Faults happen Preventing them may be technically or economically impractical
- 13. Systems can be designed and built to tolerate faults FAULT TOLERANCE
- 14. STABLE SYSTEM UNSTABLE SYSTEM
- 16. Contain the fault Using physical or logical barriers
- 17. Norlando Pobre; Flickr (www.flickr.com/photos/npobre/), CC-by
- 18. Shawn O’Neil; Flickr (www.flickr.com/photos/oneilsh/), CC-by
- 19. Pool separation Different pools for each task or dependency Brian Cantoni; Flickr (www.flickr.com/photos/cantoni/), CC-by
- 20. Asynchronous communication Sender and receiver fail independently, receiver can catch up later Synchronous communication Propagates failures, context may be lost
- 22. CLIENT ISOLATION NEW CLIENT CLIENT A CLIENT B
- 23. Avoid SPOF Find the components which compromise the system How to contain faults Invest in redundancy Improve availability by having more than one of everything Build bulkheads Set up logic walls to reduce the blast radius
- 25. A quick error is better than a slow response FAIL FAST PRINCIPLE
- 26. Validate early Anticipate problems and change course Carl Wycoff; Wikimedia Commons, CC-by
- 27. Reject politely Fast refusal is better than slow error Taber A.B.; Flickr (www.flickr.com/photos/andrewbain/), CC-by
- 28. Never wait long Protect long tasks with timeouts Robert C-B.; Flickr (www.flickr.com/photos/29233640@N07/), CC-by
- 29. Watch out for slowness In-process locks Database queries Sockets Remote APIs Albert Herring, Wikimedia Commons, CC-by
- 30. Watch out for slowness In-process locks Database queries Sockets Remote APIs Albert Herring, Wikimedia Commons, CC-by
- 31. Watch out for slowness In-process locks Database queries Sockets Remote APIs Albert Herring, Wikimedia Commons, CC-by
- 32. Watch out for slowness In-process locks Database queries Sockets Remote APIs Albert Herring, Wikimedia Commons, CC-by
- 33. Fail fast: in practice
- 35. Decline service When overloaded, ask clients to come back later How to fail fast Never wait long Set a timeout for blocking calls and slow operations Validate early Avoid starting something that cannot be completed
- 37. Circuit breakers Separate the failing parts
- 39. Fault-tolerance libraries Circuit breaking Avoid cascading failures during periods of turbulence Monitoring and alerting Observe the behaviour of all your dependencies Timeouts Time-bound any operation Fall-back Recover using an alternative path
- 41. SERVICE EVOLUTION TENANT ATENANT A TENANT ATENANT B TENANT ATENANT C
- 42. Anticipate failure If it is not going to work, do not even try How to escape Degrade gracefully A cached result or a default value may be an alternative Detect problems Compare all interactions against error thresholds
- 44. Communication Share initial expectations and status updates
- 45. Adjust Page size Growth Flow Retry Paginate queries Avoid unbounded database and remote queries
- 46. Adjust Page size Growth Flow Retry Beware of things that grow Clean up old data and set size limits
- 47. Adjust Page size Growth Flow Retry Negotiate speed Adjust dynamically using rate limits, request throttling, buffering and autoscaling
- 48. Adjust Page size Growth Flow Retry Insist smartly Retry transient errors with timeouts and back-off
- 51. Insist smartly Transient errors can be retried with back-off How to adjust Report availability Apply back pressure to prevent congestion Negotiate size Limit the cost of the job
- 53. Monitor the behaviour Define metrics, observe trends, set up alerts, capture logs Learn from experience Observe Test Reflect
- 54. Test with chaos Verify your hypothesis about fault-tolerance by continuously introducing chaos Learn from experience Observe Test Reflect
- 55. Learn from mistakes Each incident is an opportunity to make the system more robust Learn from experience Observe Test Reflect
- 59. Monitor and alert Understand the behaviour of the system in production How to learn Reflect on incidents Analyse the root cause and prevent recurrences Test what if…? Deliberately introduce chaos to assess fault-tolerance
- 61. Hope is not a strategy Test, observe, reflect Life starts after releasing “Code complete” is not “production ready” Be cynical Do not trust anybodyRobustness is an attitude
- 62. Some disasters can be prevented Build and test with failure in mind Faults are unavoidable Any possible fault will eventually happen
- 63. Leo Hidalgo; Flickr (www.flickr.com/photos/ileohidalgo/), CC-by
- 64. Release it! Michael Nygard Site Reliability Engineering Google References
- 65. Thank you! DIEGO BERRUETA | ENGINEERING PRINCIPAL | ATLASSIAN