Heroku's Ryan Smith at Waza 2013: Predictable Failure
0 likes834 views
The document discusses predictable system failures and strategies for managing them, emphasizing the necessity of understanding failure consequences. It highlights the importance of redundancy in systems while warning against its pitfalls, such as infinite regress and dormant failures. The conclusion advises that it's preferable to avoid redundancy unless necessary and to remain realistic about system reliability.
![Redundancy - Complex
func Query(conns []Conn, query string) Result {
ch := make(chan Result, len(conns))
for _, conn := range conns {
go func(c Conn) {
ch <- c.DoQuery(query):
}(conn)
}
return <-ch
}](https://image.slidesharecdn.com/predictablefailure1-130304163139-phpapp02/85/Heroku-s-Ryan-Smith-at-Waza-2013-Predictable-Failure-11-320.jpg)
Heroku's Ryan Smith at Waza 2013: Predictable Failure
- 1. Predictable Failure Building systems that fail in predictable ways.
- 2. About ● @ryandotsmith ● Engineer at Heroku for about 3 years
- 3. Things I have worked on ● Heroku API ● Messages Queues ● Usage & Billing ● DNS ● Log delivery & visibility
- 7. Your system will fail It is important to comprehend the consequences
- 9. What can you do? ● Make your apps less critical. ● Redundancy
- 10. No Redundancy - Simple func Query(c Conn, query string) Result { return c.DoQuery(query) }
- 11. Redundancy - Complex func Query(conns []Conn, query string) Result { ch := make(chan Result, len(conns)) for _, conn := range conns { go func(c Conn) { ch <- c.DoQuery(query): }(conn) } return <-ch }
- 13. A component that just works
- 14. Independence func add(x) { sys.1 return x + y// Bug! } func add(x) { sys.2 return x + y// Bug! }
- 15. Most famous bug ever
- 17. Dormant Failures
- 18. Don't wait until disaster strikes to find out that Your secondary RDMS has a full disk
- 19. Propagation
- 20. Conclusion ● Best if you don't have to build redundancy ● Consider the pitfalls if you do ○ Infinite regress ○ Indipendence ○ Dormant Failures ○ Propagation ● Don't be disillusioned 99.0 > 98.0