Fighting with scale
1 like955 views
The document discusses the complexities and challenges of building scalable systems using microservices, emphasizing the importance of choosing the right tools like Docker, Graphite, and Kafka for monitoring and messaging. It shares iterations of architecture solutions and the lessons learned from experiences with Kafka, including handling corrupt message logs and the necessity for agile responses to production traffic issues. Ultimately, it advises focus on knowhow and preparation for unexpected problems rather than relying solely on centralization.
Fighting with scale
- 1. Fighting with scale things that “Getting started” doesn’t cover Adam Dubiel
- 2. We’re building a New World
- 3. We used to have these..
- 4. ..and now we have these
- 6. Need for automation env business env business
- 7. Choosing the right tools can be hard
- 9. and then came docker..
- 17. 1st iteration
- 18. 1st iteration - overall
- 19. 1st iteration - memory
- 20. 1st iteration - cause
- 22. 2nd iteration - memory
- 23. 2nd iteration - overall
- 24. not only soft, drivers too
- 25. Metrics - StatsD
- 29. problem?
- 30. how to scale? StatsDStatsD StatsD proxy
- 31. how to scale proxy…?
- 35. but what with SLA? strict SLA, p99 < 100ms stable p999 would be great
- 36. cluster manipulations can be painful
- 37. and beware of abusive clients
- 38. 1st iteration - Kafka leaders
- 39. 1st iteration - Kafka leaders
- 40. 1st iteration - Kafka leaders Kafka does not distribute leaders uniformly simple take-first algorithm no tools to make it happen go with #!/usr/bin/python
- 41. 2nd iteration - buffering simple solution to complex problem use Kafka producer buffer we can operate without Kafka for 1h!
- 43. and when we thought we have it all covered.. bam! corrupt message logs
- 44. corrupt message logs Kafka High Level Consumer can’t skip corrupt parts no easy solution time pressure - 24h retency #!/usr/bin/python to the rescue!
- 45. Lessons learned
- 46. Production traffic is only on.. Production
- 47. prepare for the Unknown
- 48. ..but mostly for problems
- 49. scripts are your friends when time is the enemy
- 50. centralisation sounds appealing (but seriously, don’t)
- 51. focus on knowhow agility ability to create clusters on demand
- 52. isolate