Monitoring Complex Systems - Chicago Erlang, 2014
6 likes2,311 views
The document discusses the challenges and complexities of building and maintaining real-time, fault-tolerant systems with a focus on low latency and high concurrency. It emphasizes the importance of monitoring and instrumentation in understanding the performance and behavior of complex systems, highlighting the limitations of traditional verification methods. Additionally, it outlines tools like Exometer for tracking metrics and stresses the need for insights and analysis to address issues that arise in such systems.
Monitoring Complex Systems - Chicago Erlang, 2014
- 2. I do things to/with computers.
- 3. I build real-time systems.
- 4. I build fault-tolerant systems.
- 5. I build critical systems.
- 6. AdRoll
- 7. Less this.
- 8. More this.
- 9. Engineering + Mathematics = ads
- 10. Engineering + Mathematics = ads (you’re welcome)
- 11. REAL- T I M E BIDDING
- 12. The Problem Domain • Low latency ( < 100 ms per transaction)
- 13. The Problem Domain • Low latency ( < 100 ms per transaction) • Firm real-time system
- 14. The Problem Domain • Low latency ( < 100 ms per transaction) • Firm real-time system • Highly concurrent (90 billion transactions per day)
- 15. The Problem Domain • Low latency ( < 100 ms per transaction) • Firm real-time system • Highly concurrent (90 billion transactions per day) • Global, 24/7 operation
- 16. I build Complex Systems
- 17. Complex Systems • Non-linear feedback • Tightly coupled to external systems • Difficult to model, understand
- 18. Bad things happen when Complex Systems fail.
- 19. Humans are bad at predicting the performance of complex systems(…). Our ability to create large and complex systems fools us into believing that we’re also entitled to understand them. CARLOS BUENO “MATURE OPTIMIZATION HANDBOOK”
- 20. Complex Systems often create worse problems than those they solve.
- 21. The key challenge to sustaining a complex system is maintaining our understanding of it.
- 22. What can be done?
- 23. Ahead of time verification is not sufficient. ! (don’t scrimp on it, though)
- 24. Compile-time guarantees are not sufficient. ! (don’t scrimp on them, either)
- 25. We need insight into the running system.
- 26. What are we looking for? • VM killers
- 27. What are we looking for? • VM killers • Application performance regressions
- 28. What are we looking for? • VM killers • Application performance regressions • Abnormal application behavior
- 29. What are we looking for? • VM killers • Application performance regressions • Abnormal application behavior • Surprises
- 30. INSTRUMENTATION
- 31. The BEAM is ready to ride.
- 32. erlang:memory/1
- 35. What about our own work?
- 36. Exometer
- 37. Important Terms metric a measurement entry a receiver and aggregator of metrics reporter that which samples entries periodically and ships them to another system subscription the definition of the regular interval on which reporters sample entries
- 38. These are all loosely coupled at runtime.
- 39. Configuration is static, but you can adapt it on the fly.
- 40. Why exometer over the alternatives?
- 41. It is extensively documented.
- 43. It’s vigorously maintained. (Ulf Wiger fan club day.)
- 44. It is silly fast.
- 45. Okay, great. We have instrumentation.
- 46. Now what?
- 47. MONITORING
- 48. This is the hard part.
- 49. Visualization
- 50. Alerting
- 51. Analysis
- 52. Visualization tells you how things look but not why.
- 53. A good day.
- 54. Uh oh.
- 55. Growth is good, but steady is better.
- 56. Bids are stable.
- 58. What happened?
- 59. We forgot about Labor Day.
- 61. Alerting tells you that something happened, but not why.
- 62. A normal day.
- 63. Wat?
- 66. Errors prior are okay.
- 67. What happened?
- 68. “Uh, hey guys, you know Facebook is down, right?”
- 69. Analysis gives you why but only if you know how to ask for what.
- 70. The memory use of a bidder.
- 71. ಠ_ಠ
- 74. Not in ETS.
- 75. Come on now.
- 76. What happened?
- 77. A jiffy bug.
- 78. A jiffy bug. (we think)
- 79. Shout out to Miriam Pena for spending two weeks tracking this down.
- 80. Okay, great. We have monitoring and instrumentation.
- 81. Now all our problems are solved, right?
- 82. Not quite.
- 83. Instruments make up for our lack of insight.
- 84. Monitoring makes up for our frailty.
- 85. Every solution brings its own problems.
- 86. Instruments may be misleading.
- 87. Instruments may be overwhelming.
- 88. Instruments may be inaccurate.
- 89. Instruments may be ignored.
- 90. What can be done?
- 91. A little paranoia never hurt anyone.
- 93. Train.
- 94. Keep sight of the main goal.
- 95. Have resources you’re willing to sacrifice.
- 96. AdRoll is Hiring! :D
- 97. Thanks, folks! <3 @bltroutwine