TheConf 2018 - How does your app behave when everything goes wrong
1 like147 views
The document discusses various architectural patterns for building stable applications that can handle failures when communicating with external services. It describes patterns like request caching, timeouts, retries, circuit breakers, and backup batteries. Backup batteries involve storing past responses to external requests locally so the application can continue functioning even if a service is unavailable. The key is to plan for failures, measure system performance, avoid relying solely on external dependencies, focus on resilience, prioritize users, and get creative in solving problems.
TheConf 2018 - How does your app behave when everything goes wrong
- 1. How does your application behave when everything goes wrong? Emerson Macedo @emerleite
- 2. Who am I? Emerson Macedo @emerleite Software Engineer
- 9. Rio de Janeiro São Paulo&
- 10. #desapega
- 11. 630K BRAZILIANS OLX IS FOR EVERYONE Announce for the first time every month 25% OF ITEMS SOLD During first 24H
- 13. OLX Brazil – Biz Growth 3M+ Brazilians announce every month 500K+ Ads inserted every day 63% of items sold during first week 2M+ items sold per month – 50 per minute
- 15. "Anything that can go wrong will go wrong" Major Edward A. Murphy
- 24. Expectation
- 25. Reality
- 32. Your Application External Service Happy path • Body Level One • Body Level Two • Body Level Three • Body Level Four • Body Level Five
- 33. Your Application External Service Happy path • Body Level One • Body Level Two • Body Level Three • Body Level Four • Body Level Five Request 200 OK
- 34. Your Application External Service Happy path • Body Level One • Body Level Two • Body Level Three • Body Level Four • Body Level Five Request 200 OK Success
- 40. Request Cache
- 42. Your Application Cache External Service Cache read (service URI) - NOT FOUND Request Cache
- 43. Your Application Cache External Service Request Cache read (service URI) - NOT FOUND Request Cache
- 44. Your Application Cache External Service Request 200 OK Cache read (service URI) - NOT FOUND Request Cache
- 45. Your Application Cache External Service Request 200 OK Update cache (service URI) - SHORT TTL Cache read (service URI) - NOT FOUND Request Cache
- 46. Your Application Cache External Service Request 200 OK Update cache (service URI) - SHORT TTL Cache read (service URI) - NOT FOUND Request Cache Success
- 48. Your Application Cache External Service Cache read (service URI) - FOUND Request Cache
- 49. Your Application Cache External Service Cache read (service URI) - FOUND Request Cache Success
- 50. Request Timeout
- 54. Request Retry
- 65. Circuit Breaker
- 66. Circuit Breaker
- 71. Your Application State External Service Circuit Breaker Request 200 OK Circuit State - CLOSED Success
- 74. Your Application State External Service Circuit Breaker Request TIMEOUT / ERROR Circuit State - CLOSED
- 75. Your Application State External Service Circuit Breaker Request TIMEOUT / ERROR Fail Circuit State - CLOSED
- 76. Your Application State External Service Circuit Breaker Request TIMEOUT / ERROR Fail Circuit State - CLOSED Circuit State - OPEN
- 82. Your Application State External Service Circuit Breaker Request TIMEOUT / ERROR Circuit State - HALF OPEN
- 83. Your Application State External Service Circuit Breaker Request TIMEOUT / ERROR Fail Circuit State - HALF OPEN
- 84. Your Application State External Service Circuit Breaker Request TIMEOUT / ERROR Fail Circuit State - HALF OPEN Circuit State - OPEN
- 87. Your Application State External Service Circuit Breaker Request 200 OK Circuit State - HALF OPEN
- 88. Your Application State External Service Circuit Breaker Request 200 OK Success Circuit State - HALF OPEN
- 89. Your Application State External Service Circuit Breaker Request 200 OK Success Circuit State - HALF OPEN Circuit State - CLOSED
- 94. Your system must keep working
- 96. No matter the fallback strategy, it can have an impact on the business of the system Nygard, Michael - Release It
- 98. Therefore, it’s essential to involve the system’s stakeholders when deciding how to handle calls when the circuit is open Nygard, Michael - Release It
- 99. Backup Battery
- 100. A backup battery provides power to a system when the primary source of power is unavailable Wikipedia
- 106. Your Application Battery External Service Request 200 OK Charge battery (service URI) - LONG TTL Backup Battery Success
- 110. Your Application Battery External Service Request TIMEOUT / ERROR Battery response (service URI) - FOUND Backup Battery
- 111. Your Application Battery External Service Request TIMEOUT / ERROR Battery response (service URI) - FOUND Success Backup Battery
- 115. Your Application Battery External Service Request TIMEOUT / ERROR Battery response (service URI) - NOT FOUND Backup Ba4ery
- 116. Your Application Battery External Service Request TIMEOUT / ERROR Battery response (service URI) - NOT FOUND Fail Backup Ba4ery
- 121. Your Application Battery External Service Request TIMEOUT / ERROR Charger Async BATCH Backup Ba4ery
- 122. Your Application Battery External Service Request TIMEOUT / ERROR Battery response (service URI) - FOUND Charger Async BATCH Backup Ba4ery
- 123. Your Application Battery External Service Request TIMEOUT / ERROR Battery response (service URI) - FOUND SuccessCharger Async BATCH Backup Ba4ery
- 124. Backup Battery
- 126. A request cache is a trusted information stored and reused to avoid external service overload
- 127. A backup battery provides power to a system when the primary source of power is unavailable Wikipedia
- 128. A request cache should have a short time to live (a.k.a TTL)
- 129. A backup battery should have a long time to live (a.k.a TTL)
- 131. Battery Backup Examples Credit Card Limit Approval
- 132. Battery Backup Examples Credit Card Limit Approval Account Information
- 133. Battery Backup Examples Credit Card Limit Approval Account Information Listing Offer
- 134. Battery Backup Examples Credit Card Limit Approval Account Information Listing Offer Video Playback Authorization
- 135. I’ve been doing this Battery Backup as an architectural stability pattern for the last 5 years
- 136. Lessons Learned
- 138. Grafana or Similar
- 139. 2 - Do not rely on external services
- 142. 3 - Be resilient for everything
- 144. 4 - Your users are your target
- 146. 5 - Be creative and find solutions
- 153. That’s all folks Emerson Macedo @emerleite https://blog.emerleite.com