![References
● ‘Automating Failure Testing at Internet Scale [ACM SoCC’16]
https://people.ucsc.edu/~palvaro/fit-ldfi.pdf
● ‘Lineage Driven Fault Injection’ [ACM SIGMOD’15]
http://people.ucsc.edu/~palvaro/molly.pdf
● Netflix Tech Blog on ‘Automated Failure Testing’
http://techblog.netflix.com/2016/01/automated-failure-testing.html](https://image.slidesharecdn.com/orchestratedchaosreactivesummit-161011223037/85/Orchestrated-Chaos-Applying-Failure-Testing-Research-at-Scale-105-320.jpg)
![Bins and Balls
Request
Class 1
Class 2
Class 3
Class n
[...]
r’ r](https://image.slidesharecdn.com/orchestratedchaosreactivesummit-161011223037/85/Orchestrated-Chaos-Applying-Failure-Testing-Research-at-Scale-116-320.jpg)
Orchestrated Chaos: Applying Failure Testing Research at Scale.
- 1. PETER ALVARO Orchestrated Chaos With a prelude of vignettes and an appendix of fairy tales
- 2. Mythology
- 3. About me
- 4. About me
- 5. About me
- 6. Platitudes
- 9. Much harder: moving complexity around
- 10. Much harder: moving complexity around
- 11. Much harder: moving complexity around
- 12. Much harder: moving complexity around
- 13. Much harder: moving complexity around
- 14. Nontrivial systems problems always require tradeoffs Productivity / Convenience Purity / Correctness
- 15. Vignettes
- 16. Vignette 1: teaching myself docker
- 17. Vignette 2: a DBA tale
- 18. Vignette 3: selling lovely languages
- 19. Vignette 4: Microservices The UNIX philosophy: Do one thing and do it well.
- 20. > man ls
- 26. Ease of release wins
- 27. Ease of release wins
- 28. Ease of release wins
- 29. Ease of release wins
- 30. Ease of release wins
- 31. Ease of release wins
- 32. Ease of release wins
- 33. Ease of release wins
- 34. Ease of release wins
- 35. Ease of release wins
- 36. Ease of release wins
- 37. Ease of release wins
- 38. Ease of release wins
- 39. The profound solipsism of the microservice
- 40. The profound solipsism of the microservice
- 41. The profound solipsism of the microservice
- 42. The profound solipsism of the microservice
- 43. Every microservice is a piece of the continent
- 44. Every microservice is a piece of the continent
- 45. What could possibly go wrong? Consider computation involving 100 services Search Space: 2100 executions
- 46. “Depth” of bugs Single Faults Search Space: 100 executions
- 47. “Depth” of bugs Combination of 4 faults Search Space: 3M executions
- 48. “Depth” of bugs Combination of 7 faults Search Space: 16B executions
- 49. Reflections 1. Managing complexity can be a zero-sum game 2. Productivity trumps purity 3. Chaos results…. and gives rise to a new order
- 50. Opportunity
- 51. What the hell is going on? (Observability) Call graph tracing (e.g. Zipkin)
- 52. What could possibly go wrong? (Fault injection) A fault injection framework (e.g. FIT)
- 53. Random search A fault injection framework (e.g. FIT) Call graph tracing (e.g. Zipkin)
- 55. Engineer-guided search A fault injection framework (e.g. FIT) Call graph tracing (e.g. Zipkin)
- 58. A cunning malevolent sentience? A fault injection framework (e.g. FIT) Call graph tracing (e.g. Zipkin)
- 59. A cunning malevolent sentience? A fault injection framework (e.g. FIT) Call graph tracing (e.g. Zipkin)
- 60. Lineage-driven Fault Injection A fault injection framework (e.g. FIT) LDFI Call graph tracing (e.g. Zipkin)
- 61. Fault-tolerance “is just” redundancy
- 62. But how do we know redundancy when we see it? Hard question: “Could a bad thing ever happen?” Easier: “Exactly why did a good thing happen?” “What could have gone wrong?”
- 63. Lineage-driven fault injection Why did a good thing happen? Consider its lineage. The write is stable Stored on RepA Stored on RepB Bcast1 Bcast2 Client Client
- 64. Lineage-driven fault injection Why did a good thing happen? Consider its lineage. What could have gone wrong? Faults are cuts in the lineage graph. Is there a cut that breaks all supports? The write is stable Stored on RepA Stored on RepB Bcast1 Bcast2 Client Client
- 65. Lineage-driven fault injection Why did a good thing happen? Consider its lineage. What could have gone wrong? Faults are cuts in the lineage graph. Is there a cut that breaks all supports? The write is stable Stored on RepA Stored on RepB Bcast1 Bcast2 Client Client
- 66. What would have to go wrong? (RepA OR Bcast1) The write is stable Stored on RepA Stored on RepB Bcast2 Client Client Bcast1
- 67. What would have to go wrong? (RepA OR Bcast1) AND (RepA OR Bcast2) The write is stable Stored on RepA Stored on RepB Bcast1 Bcast2 Client Client
- 68. What would have to go wrong? (RepA OR Bcast1) AND (RepA OR Bcast2) AND (RepB OR Bcast2) The write is stable Stored on RepA Stored on RepB Bcast1 Client Client Bcast2
- 69. What would have to go wrong? (RepA OR Bcast1) AND (RepA OR Bcast2) AND (RepB OR Bcast2) AND (RepB OR Bcast1) The write is stable Stored on RepA Stored on RepB Bcast1 Bcast2 Client Client
- 70. Lineage-driven fault injection The write is stable Stored on RepA Stored on RepB Bcast1 Bcast2 Client Client Hypothesis: {Bcast1, Bcast2}
- 71. Lineage-driven fault injection The write is stable Stored on RepA Stored on RepB Bcast1 Bcast2 Client Client Bcast3 Client (RepA OR Bcast1) AND (RepA OR Bcast2) AND (RepB OR Bcast2) AND (RepB OR Bcast1)
- 72. Lineage-driven fault injection The write is stable Stored on RepA Stored on RepB Bcast1 Bcast2 Client Client Bcast3 Client (RepA OR Bcast1) AND (RepA OR Bcast2) AND (RepB OR Bcast2) AND (RepB OR Bcast1) AND (RepA OR Bcast3) AND (RepB OR Bcast3)
- 73. Search Space Reduction Each Experiment finds a bug, OR Reduces the Search space
- 74. Lineage-driven Fault Injection Recipe: 1. Start with a successful outcome. Work backwards. 2. Ask why it happened: Lineage 3. Convert lineage to a boolean formula and solve 4. Lather, rinse, repeat 2. Lineage 3. CNF Fail1. Success Why? Encode Solve 4. REPEAT
- 75. Minimal requirements 1. Fault injection infrastructure 2. Mechanism for collecting lineage 3. Ability to replay interactions
- 76. Lineage
- 77. Request Tracing
- 78. Request Tracing
- 81. Case study: “Netflix AppBoot” Services ~100 Search space (executions) 2100 (1,000,000,000,000,000,000,000,000,000,000) Experiments performed 200 Critical bugs found 11
- 82. Fairy tale
- 87. Growing Research Don’t: “Throw it over the wall” Do: Deep embeddings Trading shoes
- 88. Growing Research
- 89. Work with us Search prioritization Input generation Richer lineage collection
- 90. Search prioritization: minimal hitting sets (A ∨ B ∨ C ) ∧ (C ∨ D ∨ E ∨ F) ∧ (D ∨ E ∨ F ∨ G) ∧ (H ∨ I) (A, B, C), (C, D, E, F), (D, E, F, G), (H, I) ⇩
- 91. Search prioritization: minimal hitting sets (A ∨ B ∨ C ) ∧ (C ∨ D ∨ E ∨ F) ∧ (D ∨ E ∨ F ∨ G) ∧ (H ∨ I) (A, B, C), (C, D, E, F), (D, E, F, G), (H, I) ⇩ e.g. (C, E, H) ✔ X X X X X
- 92. Measuring FT by counting alternatives
- 93. Measuring fault tolerance by counting alternatives
- 94. Most likely combination of faults X X X X X
- 95. Most likely combination of faults X X X X X
- 96. Most likely combination of faults X X X X X
- 97. Input generation Using lightweight modeling to understand Chord Pamela Zave
- 98. The importance of being inputs Using lightweight modeling to understand Chord Pamela Zave
- 99. The importance of being inputs Using lightweight modeling to understand Chord Pamela Zave
- 100. The importance of being inputs Using lightweight modeling to understand Chord Pamela Zave
- 102. Where we are A fault injection framework (e.g. FIT) Call graph tracing (e.g. Zipkin)
- 103. Where we’re headed A fault injection framework (e.g. FIT) Lineage- driven fault injection Call graph tracing (e.g. Zipkin)
- 104. Thanks to our hosts, benefactors and collaborators!
- 105. References ● ‘Automating Failure Testing at Internet Scale [ACM SoCC’16] https://people.ucsc.edu/~palvaro/fit-ldfi.pdf ● ‘Lineage Driven Fault Injection’ [ACM SIGMOD’15] http://people.ucsc.edu/~palvaro/molly.pdf ● Netflix Tech Blog on ‘Automated Failure Testing’ http://techblog.netflix.com/2016/01/automated-failure-testing.html
- 106. FOLD
- 108. The profound solipsism of the microservice
- 109. UGLY
- 110. GOOD RAW
- 111. GOOD RAW
- 112. GOOD RAW
- 113. GOOD RAW
- 114. True Silicon Valley Stories 1. Crazy legwork 2. The “what the hell does our site do” project 3. Offsite => online
- 115. Replay
- 116. Bins and Balls Request Class 1 Class 2 Class 3 Class n [...] r’ r
- 117. Class n Predicting Request Graphs Request
- 118. Class n Predicting Request Graphs Request Some function f: Requests → Classes
- 119. F( ) = Class n Request Predicting Request Graphs
- 120. The profound solipsism of the microservice