Reliability and Resilience
- 1. Thank you to our Sponsors Media Sponsor:
- 2. System Reliability and Resilience and stuff
- 3. tuple
- 4. //Initialize customer and invoice Initialize(customer, invoice);
- 5. public void Initialize (Customer customer, Invoice invoice) { customer.Name = “asdf”; invoice.Date = DateTime.Now; }
- 6. Initialize(customer, invoice); //did something happen to customer // and/or invoice?
- 7. customer.Name = InitNameFrom(customer, invoice); invoice.Date = InitDateFrom(customer, invoice);
- 8. customer.Name = GetNameFrom(customer, invoice); invoice.Date = GetDateFrom(customer, invoice);
- 9. var results = Initialize(customer, invoice); customer.Name = results.Item1; invoice.Date = results.Item2;
- 10. public tuple<string, DateTime> Initialize(customer, invoice) { return new Tuple<string, DateTime> (“asdf”, DateTime.Now); }
- 11. public static bool TryParse (string s, out DateTime result) or public static tuple<bool, DateTime?> TryParse (string s)
- 12. tuple • Avoid side effects • Avoid out parameters • multiple values without a specific type
- 13. null object
- 14. private ILogger _logger; public MyClass(ILogger logger) { _logger = logger; } … if (_logger != null) { _logger.Debug( “it worked on my machine!”); }
- 15. null checks for everyone!
- 16. forget one and…
- 17. public class NullLogger : ILogger { public void Debug(string text) { //do sweet nothing } }
- 18. private ILogger _logger = new NullLogger(); public MyClass(ILogger logger) { _logger = logger; } … _logger.Debug( “it worked on my machine!”);
- 19. null object • Can eliminate null checks • Simple to implement
- 20. Circuit Breaker
- 22. Retry
- 23. Out of Process Dependency N times Your Application
- 24. Out of Process Dependency N times * Y clients
- 25. = Denial of Service Attack
- 26. Limit the # of retries
- 27. N * Y becomes 5 * Y
- 28. Y is still a problem
- 30. Circuit Breaker
- 32. State Machine On :: Off
- 33. On Off when not healthy
- 34. Off On manually
- 35. Get to software before we ask you to dance
- 36. Out of Process Dependency Healthy or Unhealthy
- 37. State is independent of requestor Out of Process Dependency
- 38. Your Application Has many independent external dependencies
- 39. Your Application Can throttle itself
- 40. Your Application Has a wait threshold
- 41. Your Application External Dependency Circuit Breaker Threshold = 2 Pause = 10ms Timeout = 30s Request State = Closed Request Failure (i.e. HTTP 500) Failure Count = 1 Pause 10ms Request Failure (i.e. HTTP 500) Failure Count = 2 State = Open OperationFailedException
- 42. Threshold = 2 Pause = 10ms Timeout = 30s Request State = Open 30s has not passed CircuitBreakerOpenException Request 30s has not passed CircuitBreakerOpenException System can try to become healthy for 30s Your Application External Dependency Circuit Breaker
- 43. Threshold = 2 Pause = 10ms Timeout = 30s 30s has passed Your Application Request State = ½ Open Request Failure (i.e. HTTP 500) Failure Count = 2 State = Open OperationFailedException External Dependency Circuit Breaker
- 44. Threshold = 2 Pause = 10ms Timeout = 30s 30s has passed Request State = ½ Open Request Failure Count = 0 State = Closed Response Response Your Application External Dependency Circuit Breaker
- 45. Closed Open ½ Open
- 46. ½ Open is like a manual reset
- 47. Pause Timeout
- 48. Pause between calls in the loop
- 49. Timeout before you can call again
- 50. Exceptions
- 53. Don’t Loose Exception Info
- 55. Threshold = 3 Request State = Closed Request Failure (i.e. HTTP 500) ? Request Request Failure (i.e. HTTP 500) Failure Count = 2 Failure Count = 0 State = Closed Response Response Your Application External Dependency Circuit Breaker Failure Count = 1
- 58. Dependency type, endpoint svc, endpoint
- 59. Where?
- 60. Your Application Circuit Breaker Proxy Out of Process Dependency
- 62. Your Application Circuit Breaker Proxy Web Service Circuit Breaker Repository Database
- 63. circuit breaker • retry looping • slow down attempts • good neighbour
- 64. Thank you Donald Belcham @dbelcham donald.belcham@igloocoder.com